2023-10-13T23:51:55.438 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-13T23:51:55.439 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_dis@teuthology --archive /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274 --description rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} --name dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi -- /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274/orig.config.yaml 2023-10-13T23:51:55.473 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-13T23:51:55.545 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274 branch: wip-dis-testing description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} email: idryomov@gmail.com first_in_suite: false job_id: '7427274' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: rbd default data pool: datapool rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-qa sha1: 735f68d7bf91d643813a8420fc2b11441f4bd8c4 owner: scheduled_dis@teuthology priority: 74 repo: https://github.com/ceph/ceph-ci roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2449 sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f sleep_before_teardown: 0 subset: 49/99 suite: rbd suite_branch: wip-dis-testing-qa suite_path: /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa suite_relpath: qa suite_repo: https://github.com/idryomov/ceph suite_sha1: 735f68d7bf91d643813a8420fc2b11441f4bd8c4 targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYvQckp2HFA+3Xt5famVR0+ItJimBVjM4uj2TjVxZTAVlb8a2GoWX8NXDIwcbxdsG5JcnrW4QgTnV7/jDpR5BU= tasks: - install: null - ceph: null - exec: client.0: - sudo ceph osd pool create datapool 4 - rbd pool init datapool - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-13_23:35:38 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-13T23:51:55.545 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa; will attempt to use it 2023-10-13T23:51:55.546 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks 2023-10-13T23:51:55.546 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-13T23:51:55.548 INFO:teuthology.task.internal:Checking packages... 2023-10-13T23:51:55.573 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '01fff6a72a328459c1d153e5dc1de6a34e48a82f' 2023-10-13T23:51:55.573 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-13T23:51:55.573 INFO:teuthology.packaging:ref: None 2023-10-13T23:51:55.574 INFO:teuthology.packaging:tag: None 2023-10-13T23:51:55.574 INFO:teuthology.packaging:branch: wip-dis-testing 2023-10-13T23:51:55.574 INFO:teuthology.packaging:sha1: 01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-13T23:51:55.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing 2023-10-13T23:51:55.715 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6724-g01fff6a7-1jammy 2023-10-13T23:51:55.716 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-13T23:51:55.724 INFO:teuthology.task.internal:no buildpackages task found 2023-10-13T23:51:55.724 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-13T23:51:55.732 INFO:teuthology.task.internal:Saving configuration 2023-10-13T23:51:55.746 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-13T23:51:55.752 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-13T23:51:55.778 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-13 23:46:11.581013', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPoDVdf/U6xGFw3c4F3AzXE7zXuSOYXxbkGt4fzJMAP8'} 2023-10-13T23:51:55.779 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-13T23:51:55.786 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-13T23:51:55.787 INFO:teuthology.run_tasks:Running task console_log... 2023-10-13T23:51:55.828 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcce12dd940>, signals=[15]) 2023-10-13T23:51:55.829 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-13T23:51:55.837 INFO:teuthology.task.internal:Opening connections... 2023-10-13T23:51:55.837 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2023-10-13T23:51:55.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:51:55.917 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-13T23:51:55.924 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2023-10-13T23:51:56.026 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2023-10-13T23:51:56.027 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2023-10-13T23:51:56.074 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-13T23:51:56.075 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Ubuntu" 2023-10-13T23:51:56.075 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="22.04" 2023-10-13T23:51:56.075 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-13T23:51:56.075 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_CODENAME=jammy 2023-10-13T23:51:56.075 INFO:teuthology.orchestra.run.smithi105.stdout:ID=ubuntu 2023-10-13T23:51:56.076 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE=debian 2023-10-13T23:51:56.076 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-13T23:51:56.076 INFO:teuthology.orchestra.run.smithi105.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-13T23:51:56.076 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-13T23:51:56.076 INFO:teuthology.orchestra.run.smithi105.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-13T23:51:56.077 INFO:teuthology.orchestra.run.smithi105.stdout:UBUNTU_CODENAME=jammy 2023-10-13T23:51:56.077 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2023-10-13T23:51:56.098 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-13T23:51:56.108 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-13T23:51:56.123 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-13T23:51:56.123 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2023-10-13T23:51:56.128 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-13T23:51:56.136 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-13T23:51:56.136 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2023-10-13T23:51:56.176 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-13T23:51:56.209 INFO:teuthology.run_tasks:Running task kernel... 2023-10-13T23:51:56.225 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-13T23:51:56.226 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-13T23:51:56.226 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-13T23:51:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-13T23:51:56.232 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2023-10-13T23:51:56.278 INFO:teuthology.orchestra.run.smithi105.stdout:5.15.0-56-generic 2023-10-13T23:51:56.279 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.15.0-56-generic 2023-10-13T23:51:56.279 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get clean 2023-10-13T23:51:56.418 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2023-10-13T23:51:56.565 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:51:56.589 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-13T23:51:56.658 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-13T23:51:56.702 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-13T23:51:56.834 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [506 kB] 2023-10-13T23:51:56.910 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,062 kB] 2023-10-13T23:51:56.958 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [234 kB] 2023-10-13T23:51:56.972 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-13T23:51:56.973 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [974 kB] 2023-10-13T23:51:57.000 INFO:teuthology.orchestra.run.smithi105.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.3 kB] 2023-10-13T23:51:57.000 INFO:teuthology.orchestra.run.smithi105.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [157 kB] 2023-10-13T23:51:57.041 INFO:teuthology.orchestra.run.smithi105.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [532 B] 2023-10-13T23:51:57.042 INFO:teuthology.orchestra.run.smithi105.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [991 kB] 2023-10-13T23:51:57.051 INFO:teuthology.orchestra.run.smithi105.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [658 kB] 2023-10-13T23:51:57.087 INFO:teuthology.orchestra.run.smithi105.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [217 kB] 2023-10-13T23:51:57.090 INFO:teuthology.orchestra.run.smithi105.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-13T23:51:57.091 INFO:teuthology.orchestra.run.smithi105.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-13T23:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-13T23:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-13T23:51:57.092 INFO:teuthology.orchestra.run.smithi105.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-13T23:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [33.9 kB] 2023-10-13T23:51:57.097 INFO:teuthology.orchestra.run.smithi105.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [41.7 kB] 2023-10-13T23:51:57.098 INFO:teuthology.orchestra.run.smithi105.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-13T23:51:57.122 INFO:teuthology.orchestra.run.smithi105.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-13T23:51:57.123 INFO:teuthology.orchestra.run.smithi105.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [24.3 kB] 2023-10-13T23:51:57.125 INFO:teuthology.orchestra.run.smithi105.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [13.4 kB] 2023-10-13T23:51:57.126 INFO:teuthology.orchestra.run.smithi105.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-13T23:51:57.126 INFO:teuthology.orchestra.run.smithi105.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [640 B] 2023-10-13T23:51:57.153 INFO:teuthology.orchestra.run.smithi105.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [346 kB] 2023-10-13T23:51:57.413 INFO:teuthology.orchestra.run.smithi105.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [856 kB] 2023-10-13T23:51:57.556 INFO:teuthology.orchestra.run.smithi105.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [175 kB] 2023-10-13T23:51:57.576 INFO:teuthology.orchestra.run.smithi105.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-13T23:51:57.578 INFO:teuthology.orchestra.run.smithi105.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [953 kB] 2023-10-13T23:51:57.641 INFO:teuthology.orchestra.run.smithi105.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-13T23:51:57.642 INFO:teuthology.orchestra.run.smithi105.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [154 kB] 2023-10-13T23:51:57.652 INFO:teuthology.orchestra.run.smithi105.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [532 B] 2023-10-13T23:51:57.652 INFO:teuthology.orchestra.run.smithi105.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [789 kB] 2023-10-13T23:51:57.687 INFO:teuthology.orchestra.run.smithi105.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [560 kB] 2023-10-13T23:51:57.869 INFO:teuthology.orchestra.run.smithi105.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-13T23:51:58.250 INFO:teuthology.orchestra.run.smithi105.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.7 kB] 2023-10-13T23:51:58.257 INFO:teuthology.orchestra.run.smithi105.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-13T23:51:58.273 INFO:teuthology.orchestra.run.smithi105.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-13T23:51:58.273 INFO:teuthology.orchestra.run.smithi105.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-13T23:51:58.276 INFO:teuthology.orchestra.run.smithi105.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-13T23:52:02.628 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 9,489 kB in 2s (5,330 kB/s) 2023-10-13T23:52:03.960 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-13T23:52:03.976 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-13T23:52:04.038 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-13T23:52:04.222 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2023-10-13T23:52:04.223 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2023-10-13T23:52:04.380 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:52:04.380 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:52:04.381 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:52:04.381 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:52:04.381 INFO:teuthology.orchestra.run.smithi105.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-13T23:52:04.382 INFO:teuthology.orchestra.run.smithi105.stdout: python2.7-minimal 2023-10-13T23:52:04.382 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:52:04.382 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2023-10-13T23:52:04.383 INFO:teuthology.orchestra.run.smithi105.stdout: linux-generic linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-13T23:52:04.383 INFO:teuthology.orchestra.run.smithi105.stdout: linux-headers-generic linux-image-5.15.0-86-generic 2023-10-13T23:52:04.383 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-5.15.0-86-generic linux-modules-extra-5.15.0-86-generic 2023-10-13T23:52:04.384 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2023-10-13T23:52:04.384 INFO:teuthology.orchestra.run.smithi105.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-13T23:52:04.384 INFO:teuthology.orchestra.run.smithi105.stdout:Recommended packages: 2023-10-13T23:52:04.384 INFO:teuthology.orchestra.run.smithi105.stdout: thermald 2023-10-13T23:52:04.436 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2023-10-13T23:52:04.437 INFO:teuthology.orchestra.run.smithi105.stdout: linux-headers-5.15.0-86 linux-headers-5.15.0-86-generic 2023-10-13T23:52:04.437 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.15.0-86-generic linux-modules-5.15.0-86-generic 2023-10-13T23:52:04.437 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-extra-5.15.0-86-generic 2023-10-13T23:52:04.438 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages will be upgraded: 2023-10-13T23:52:04.438 INFO:teuthology.orchestra.run.smithi105.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-13T23:52:04.490 INFO:teuthology.orchestra.run.smithi105.stdout:3 upgraded, 5 newly installed, 0 to remove and 268 not upgraded. 2023-10-13T23:52:04.491 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 113 MB of archives. 2023-10-13T23:52:04.491 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-13T23:52:04.491 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-86-generic amd64 5.15.0-86.96 [22.4 MB] 2023-10-13T23:52:04.904 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-86-generic amd64 5.15.0-86.96 [11.5 MB] 2023-10-13T23:52:04.992 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-86-generic amd64 5.15.0-86.96 [63.9 MB] 2023-10-13T23:52:05.788 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.86.83 [1,702 B] 2023-10-13T23:52:05.788 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.86.83 [2,472 B] 2023-10-13T23:52:05.788 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86 all 5.15.0-86.96 [12.3 MB] 2023-10-13T23:52:05.906 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-86-generic amd64 5.15.0-86.96 [2,882 kB] 2023-10-13T23:52:05.927 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.86.83 [2,328 B] 2023-10-13T23:52:06.313 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 113 MB in 1s (76.3 MB/s) 2023-10-13T23:52:06.490 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-modules-5.15.0-86-generic. 2023-10-13T23:52:08.703 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-13T23:52:08.708 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../0-linux-modules-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:09.045 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:11.254 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-image-5.15.0-86-generic. 2023-10-13T23:52:11.275 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../1-linux-image-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:11.363 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:11.841 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-86-generic. 2023-10-13T23:52:11.861 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:11.924 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:19.067 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../3-linux-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:52:19.208 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:52:19.523 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:52:19.645 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-image-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:52:19.965 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-headers-5.15.0-86. 2023-10-13T23:52:19.987 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../5-linux-headers-5.15.0-86_5.15.0-86.96_all.deb ... 2023-10-13T23:52:20.031 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-13T23:52:25.581 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-headers-5.15.0-86-generic. 2023-10-13T23:52:25.618 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../6-linux-headers-5.15.0-86-generic_5.15.0-86.96_amd64.deb ... 2023-10-13T23:52:25.656 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:28.476 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.86.83_amd64.deb ... 2023-10-13T23:52:28.626 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-headers-generic (5.15.0.86.83) over (5.15.0.56.54) ... 2023-10-13T23:52:28.992 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-headers-5.15.0-86 (5.15.0-86.96) ... 2023-10-13T23:52:29.114 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-headers-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:29.240 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-headers-generic (5.15.0.86.83) ... 2023-10-13T23:52:29.384 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:30.711 INFO:teuthology.orchestra.run.smithi105.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:52:30.712 INFO:teuthology.orchestra.run.smithi105.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-13T23:52:30.712 INFO:teuthology.orchestra.run.smithi105.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:52:30.712 INFO:teuthology.orchestra.run.smithi105.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-86-generic 2023-10-13T23:52:30.882 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-modules-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:32.705 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-modules-extra-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:34.115 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-image-generic (5.15.0.86.83) ... 2023-10-13T23:52:34.241 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-generic (5.15.0.86.83) ... 2023-10-13T23:52:34.367 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for linux-image-5.15.0-86-generic (5.15.0-86.96) ... 2023-10-13T23:52:34.463 INFO:teuthology.orchestra.run.smithi105.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-13T23:52:34.463 INFO:teuthology.orchestra.run.smithi105.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:52:49.386 INFO:teuthology.orchestra.run.smithi105.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-13T23:52:49.386 INFO:teuthology.orchestra.run.smithi105.stdout:Sourcing file `/etc/default/grub' 2023-10-13T23:52:49.419 INFO:teuthology.orchestra.run.smithi105.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-13T23:52:49.449 INFO:teuthology.orchestra.run.smithi105.stdout:Generating grub configuration file ... 2023-10-13T23:52:49.721 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:52:49.745 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:52:50.063 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:52:50.068 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-13T23:52:50.137 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-13T23:52:50.143 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-13T23:52:50.342 INFO:teuthology.orchestra.run.smithi105.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-13T23:52:50.343 INFO:teuthology.orchestra.run.smithi105.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-13T23:52:50.343 INFO:teuthology.orchestra.run.smithi105.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-13T23:52:50.383 INFO:teuthology.orchestra.run.smithi105.stdout:done 2023-10-13T23:52:50.890 DEBUG:teuthology.orchestra.run.smithi105:> dpkg -s linux-image-generic 2023-10-13T23:52:50.959 INFO:teuthology.orchestra.run.smithi105.stdout:Package: linux-image-generic 2023-10-13T23:52:50.959 INFO:teuthology.orchestra.run.smithi105.stdout:Status: install ok installed 2023-10-13T23:52:50.959 INFO:teuthology.orchestra.run.smithi105.stdout:Priority: optional 2023-10-13T23:52:50.959 INFO:teuthology.orchestra.run.smithi105.stdout:Section: kernel 2023-10-13T23:52:50.960 INFO:teuthology.orchestra.run.smithi105.stdout:Installed-Size: 20 2023-10-13T23:52:50.960 INFO:teuthology.orchestra.run.smithi105.stdout:Maintainer: Ubuntu Kernel Team 2023-10-13T23:52:50.960 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: amd64 2023-10-13T23:52:50.960 INFO:teuthology.orchestra.run.smithi105.stdout:Source: linux-meta 2023-10-13T23:52:50.960 INFO:teuthology.orchestra.run.smithi105.stdout:Version: 5.15.0.86.83 2023-10-13T23:52:50.961 INFO:teuthology.orchestra.run.smithi105.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-13T23:52:50.961 INFO:teuthology.orchestra.run.smithi105.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-13T23:52:50.961 INFO:teuthology.orchestra.run.smithi105.stdout:Recommends: thermald 2023-10-13T23:52:50.961 INFO:teuthology.orchestra.run.smithi105.stdout:Description: Generic Linux kernel image 2023-10-13T23:52:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: This package will always depend on the latest generic kernel image 2023-10-13T23:52:50.962 INFO:teuthology.orchestra.run.smithi105.stdout: available. 2023-10-13T23:52:50.962 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-86-generic 2023-10-13T23:52:50.962 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-13T23:52:50.962 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-13T23:52:50.963 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-13T23:52:50.963 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2023-10-13T23:52:50.963 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get clean 2023-10-13T23:52:51.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2023-10-13T23:52:51.259 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-13T23:52:51.267 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-13T23:52:51.284 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-13T23:52:51.381 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-13T23:52:52.466 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-13T23:52:52.483 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-13T23:52:52.534 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-13T23:52:52.703 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2023-10-13T23:52:52.704 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2023-10-13T23:52:52.844 INFO:teuthology.orchestra.run.smithi105.stdout:linux-image-generic is already the newest version (5.15.0.86.83). 2023-10-13T23:52:52.844 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2023-10-13T23:52:52.844 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-13T23:52:52.845 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-13T23:52:52.845 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-13T23:52:52.845 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-13T23:52:52.845 INFO:teuthology.orchestra.run.smithi105.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-13T23:52:52.845 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-13T23:52:52.878 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 0 newly installed, 0 to remove and 268 not upgraded. 2023-10-13T23:52:52.880 DEBUG:teuthology.orchestra.run.smithi105:> dpkg -s linux-image-generic 2023-10-13T23:52:52.936 INFO:teuthology.orchestra.run.smithi105.stdout:Package: linux-image-generic 2023-10-13T23:52:52.936 INFO:teuthology.orchestra.run.smithi105.stdout:Status: install ok installed 2023-10-13T23:52:52.937 INFO:teuthology.orchestra.run.smithi105.stdout:Priority: optional 2023-10-13T23:52:52.937 INFO:teuthology.orchestra.run.smithi105.stdout:Section: kernel 2023-10-13T23:52:52.937 INFO:teuthology.orchestra.run.smithi105.stdout:Installed-Size: 20 2023-10-13T23:52:52.937 INFO:teuthology.orchestra.run.smithi105.stdout:Maintainer: Ubuntu Kernel Team 2023-10-13T23:52:52.937 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: amd64 2023-10-13T23:52:52.938 INFO:teuthology.orchestra.run.smithi105.stdout:Source: linux-meta 2023-10-13T23:52:52.938 INFO:teuthology.orchestra.run.smithi105.stdout:Version: 5.15.0.86.83 2023-10-13T23:52:52.938 INFO:teuthology.orchestra.run.smithi105.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-86), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-13T23:52:52.938 INFO:teuthology.orchestra.run.smithi105.stdout:Depends: linux-image-5.15.0-86-generic, linux-modules-extra-5.15.0-86-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-13T23:52:52.938 INFO:teuthology.orchestra.run.smithi105.stdout:Recommends: thermald 2023-10-13T23:52:52.939 INFO:teuthology.orchestra.run.smithi105.stdout:Description: Generic Linux kernel image 2023-10-13T23:52:52.939 INFO:teuthology.orchestra.run.smithi105.stdout: This package will always depend on the latest generic kernel image 2023-10-13T23:52:52.939 INFO:teuthology.orchestra.run.smithi105.stdout: available. 2023-10-13T23:52:52.939 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2023-10-13T23:52:52.981 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.Z7RfpEFgLz 2023-10-13T23:52:52.982 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Z7RfpEFgLz 2023-10-13T23:52:53.050 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.Z7RfpEFgLz 2023-10-13T23:52:53.165 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.Z7RfpEFgLz is 10KB 2023-10-13T23:52:53.177 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.Z7RfpEFgLz 2023-10-13T23:52:53.183 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-13T23:52:53.237 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-13T23:52:53.237 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-13T23:52:53.238 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-13T23:52:53.304 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-86-generic 2023-10-13T23:52:53.305 DEBUG:teuthology.orchestra.run.smithi105:> sudo update-grub 2023-10-13T23:52:54.572 INFO:teuthology.orchestra.run.smithi105.stderr:Sourcing file `/etc/default/grub' 2023-10-13T23:52:54.573 INFO:teuthology.orchestra.run.smithi105.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-13T23:52:54.586 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2023-10-13T23:52:54.810 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.15.0-86-generic 2023-10-13T23:52:54.816 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.15.0-86-generic 2023-10-13T23:52:55.064 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-13T23:52:55.069 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-13T23:52:55.108 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-13T23:52:55.113 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-13T23:52:55.218 INFO:teuthology.orchestra.run.smithi105.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-13T23:52:55.219 INFO:teuthology.orchestra.run.smithi105.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-13T23:52:55.219 INFO:teuthology.orchestra.run.smithi105.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-13T23:52:55.235 INFO:teuthology.orchestra.run.smithi105.stderr:done 2023-10-13T23:52:55.236 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2023-10-13T23:53:25.266 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-13T23:53:25.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-13T23:53:25.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:53:43.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2023-10-13T23:53:52.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-13T23:53:52.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:53:55.866 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2023-10-13T23:54:07.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-13T23:54:07.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:54:26.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2023-10-13T23:54:41.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-13T23:54:41.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-13T23:54:42.479 DEBUG:teuthology.orchestra.run.smithi105:> true 2023-10-13T23:54:43.140 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-13T23:54:43.141 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-86-generic"... 2023-10-13T23:54:43.141 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2023-10-13T23:54:43.146 INFO:teuthology.orchestra.run.smithi105.stdout:5.15.0-86-generic 2023-10-13T23:54:43.146 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-86-generic vs 5.15.0-86-generic 2023-10-13T23:54:43.147 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-13T23:54:43.147 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-13T23:54:44.148 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-13T23:54:44.149 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-13T23:54:44.262 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2023-10-13T23:54:44.285 DEBUG:teuthology.parallel:result is None 2023-10-13T23:54:44.286 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-13T23:54:44.295 INFO:teuthology.task.internal:Creating test directory... 2023-10-13T23:54:44.295 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-13T23:54:44.301 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-13T23:54:44.314 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-13T23:54:44.354 INFO:teuthology.task.internal:Creating archive directory... 2023-10-13T23:54:44.354 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-13T23:54:44.377 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-13T23:54:44.387 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-13T23:54:44.388 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-13T23:54:44.435 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-13T23:54:44.445 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-13T23:54:44.446 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-13T23:54:44.457 INFO:teuthology.task.internal:Configuring sudo... 2023-10-13T23:54:44.458 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-13T23:54:44.501 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-13T23:54:44.510 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-13T23:54:44.511 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-13T23:54:44.547 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-13T23:54:44.595 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-13T23:54:44.643 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-13T23:54:44.644 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-13T23:54:44.699 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2023-10-13T23:54:44.826 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-13T23:54:44.834 INFO:teuthology.task.internal:Starting timer... 2023-10-13T23:54:44.835 INFO:teuthology.run_tasks:Running task pcp... 2023-10-13T23:54:44.846 INFO:teuthology.run_tasks:Running task selinux... 2023-10-13T23:54:44.858 DEBUG:teuthology.task.selinux:Excluding smithi105: OS 'ubuntu' does not support SELinux 2023-10-13T23:54:44.859 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-13T23:54:44.859 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-13T23:54:44.859 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-13T23:54:44.859 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-13T23:54:44.873 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-13T23:54:44.874 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-13T23:54:44.889 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-13T23:54:44.890 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-14T00:02:29.941 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2023-10-14T00:02:29.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-14T00:02:29.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T00:02:30.023 DEBUG:teuthology.orchestra.run.smithi105:> true 2023-10-14T00:02:30.142 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-10-14T00:02:30.143 INFO:teuthology.run_tasks:Running task clock... 2023-10-14T00:02:30.172 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-14T00:02:30.172 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-14T00:02:30.173 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T00:02:30.208 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-14T00:02:30.209 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Command line: ntpd -gq 2023-10-14T00:02:30.209 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: ---------------------------------------------------- 2023-10-14T00:02:30.209 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: ntp-4 is maintained by Network Time Foundation, 2023-10-14T00:02:30.209 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-14T00:02:30.210 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: corporation. Support and training for ntp-4 are 2023-10-14T00:02:30.210 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: available at https://www.nwtime.org/support 2023-10-14T00:02:30.210 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: ---------------------------------------------------- 2023-10-14T00:02:30.211 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: proto: precision = 0.044 usec (-24) 2023-10-14T00:02:30.211 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: basedate set to 2022-02-04 2023-10-14T00:02:30.211 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: gps base set to 2022-02-06 (week 2196) 2023-10-14T00:02:30.212 INFO:teuthology.orchestra.run.smithi105.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-14T00:02:30.212 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-14T00:02:30.212 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: restrict ::: KOD does nothing without LIMITED. 2023-10-14T00:02:30.213 INFO:teuthology.orchestra.run.smithi105.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-14T00:02:30.214 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listen and drop on 0 v6wildcard [::]:123 2023-10-14T00:02:30.214 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-14T00:02:30.214 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listen normally on 2 lo 127.0.0.1:123 2023-10-14T00:02:30.214 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listen normally on 3 enp3s0f1 172.21.15.105:123 2023-10-14T00:02:30.215 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listen normally on 4 lo [::1]:123 2023-10-14T00:02:30.215 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d429%5]:123 2023-10-14T00:02:30.215 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:30 ntpd[28038]: Listening on routing socket on fd #22 for interface updates 2023-10-14T00:02:31.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:31.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:31.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:31.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:31.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:31.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:31.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:31.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:31 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:32.209 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:32 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:32.209 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:32 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:32 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:32 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:32 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:32.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:32 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:33.209 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:33 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:33.209 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:33 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:33.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:33 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:33.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:33 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:33.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:33 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:33.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:33 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:34.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:34 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:34.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:34 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:34.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:34 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:34.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:34 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:34.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:34 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:34.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:34 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:35.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:35.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:35.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:35.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:35.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:35.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:35.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:35.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:35.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:35.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:35.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:35.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:35 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:36.209 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:36 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:36.209 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:36 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:36.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:36 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:36.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:36 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:36.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:36 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:36.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:36 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:37.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:37.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:37.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:37.210 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:37.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:37.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:37.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:37.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:37.211 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-14T00:02:37.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/rawstats.20231014: Permission denied 2023-10-14T00:02:37.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-14T00:02:37.212 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/peerstats.20231014: Permission denied 2023-10-14T00:02:37.212 INFO:teuthology.orchestra.run.smithi105.stdout:14 Oct 00:02:37 ntpd[28038]: ntpd: time slew -0.000790 s 2023-10-14T00:02:37.213 INFO:teuthology.orchestra.run.smithi105.stdout:ntpd: time slew -0.000790s 2023-10-14T00:02:37.213 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-14T00:02:37.213 INFO:teuthology.orchestra.run.smithi105.stderr:14 Oct 00:02:37 ntpd[28038]: can't open /var/log/ntpstats/loopstats.20231014: Permission denied 2023-10-14T00:02:37.372 INFO:teuthology.orchestra.run.smithi105.stdout: remote refid st t when poll reach delay offset jitter 2023-10-14T00:02:37.373 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================== 2023-10-14T00:02:37.373 INFO:teuthology.orchestra.run.smithi105.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:37.373 INFO:teuthology.orchestra.run.smithi105.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:37.373 INFO:teuthology.orchestra.run.smithi105.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:37.373 INFO:teuthology.orchestra.run.smithi105.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-14T00:02:37.375 INFO:teuthology.run_tasks:Running task install... 2023-10-14T00:02:37.408 DEBUG:teuthology.task.install:project ceph 2023-10-14T00:02:37.408 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}} 2023-10-14T00:02:37.409 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'} 2023-10-14T00:02:37.409 INFO:teuthology.task.install:Using flavor: default 2023-10-14T00:02:37.415 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-14T00:02:37.415 INFO:teuthology.task.install:extra packages: [] 2023-10-14T00:02:37.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-key list | grep Ceph 2023-10-14T00:02:37.479 INFO:teuthology.orchestra.run.smithi105.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-14T00:02:37.510 INFO:teuthology.orchestra.run.smithi105.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-14T00:02:37.510 INFO:teuthology.orchestra.run.smithi105.stdout:uid [ unknown] Ceph.com (release key) 2023-10-14T00:02:37.511 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-14T00:02:37.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:02:37.654 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default/ 2023-10-14T00:02:37.655 INFO:teuthology.task.install.deb:Package version is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:02:37.823 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:02:37.824 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-14T00:02:37.838 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2023-10-14T00:02:38.000 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-14T00:02:38.121 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-14T00:02:38.122 INFO:teuthology.orchestra.run.smithi105.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy InRelease 2023-10-14T00:02:38.159 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-14T00:02:38.197 INFO:teuthology.orchestra.run.smithi105.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-14T00:02:38.202 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-14T00:02:38.233 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-10-14T00:02:38.276 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2023-10-14T00:02:38.283 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-14T00:02:38.414 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 28.5 kB in 0s (70.6 kB/s) 2023-10-14T00:02:39.292 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-14T00:02:39.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6724-g01fff6a7-1jammy cephadm=18.0.0-6724-g01fff6a7-1jammy ceph-mds=18.0.0-6724-g01fff6a7-1jammy ceph-mgr=18.0.0-6724-g01fff6a7-1jammy ceph-common=18.0.0-6724-g01fff6a7-1jammy ceph-fuse=18.0.0-6724-g01fff6a7-1jammy ceph-test=18.0.0-6724-g01fff6a7-1jammy ceph-volume=18.0.0-6724-g01fff6a7-1jammy radosgw=18.0.0-6724-g01fff6a7-1jammy python3-rados=18.0.0-6724-g01fff6a7-1jammy python3-rgw=18.0.0-6724-g01fff6a7-1jammy python3-cephfs=18.0.0-6724-g01fff6a7-1jammy python3-rbd=18.0.0-6724-g01fff6a7-1jammy libcephfs2=18.0.0-6724-g01fff6a7-1jammy libcephfs-dev=18.0.0-6724-g01fff6a7-1jammy librados2=18.0.0-6724-g01fff6a7-1jammy librbd1=18.0.0-6724-g01fff6a7-1jammy rbd-fuse=18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:02:39.359 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-14T00:02:39.522 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2023-10-14T00:02:39.523 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2023-10-14T00:02:39.672 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2023-10-14T00:02:39.672 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-14T00:02:39.672 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-14T00:02:39.673 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-14T00:02:39.673 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-14T00:02:39.673 INFO:teuthology.orchestra.run.smithi105.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-14T00:02:39.673 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-14T00:02:39.675 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2023-10-14T00:02:39.675 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-14T00:02:39.675 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-14T00:02:39.675 INFO:teuthology.orchestra.run.smithi105.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-14T00:02:39.676 INFO:teuthology.orchestra.run.smithi105.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-14T00:02:39.676 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-14T00:02:39.676 INFO:teuthology.orchestra.run.smithi105.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-14T00:02:39.676 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-14T00:02:39.676 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-14T00:02:39.677 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-10-14T00:02:39.677 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-14T00:02:39.677 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-14T00:02:39.677 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-14T00:02:39.678 INFO:teuthology.orchestra.run.smithi105.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-14T00:02:39.678 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-14T00:02:39.678 INFO:teuthology.orchestra.run.smithi105.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-14T00:02:39.678 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-10-14T00:02:39.679 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-10-14T00:02:39.679 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-14T00:02:39.679 INFO:teuthology.orchestra.run.smithi105.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-14T00:02:39.679 INFO:teuthology.orchestra.run.smithi105.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-14T00:02:39.679 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-14T00:02:39.680 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2023-10-14T00:02:39.680 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2023-10-14T00:02:39.680 INFO:teuthology.orchestra.run.smithi105.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-14T00:02:39.680 INFO:teuthology.orchestra.run.smithi105.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-14T00:02:39.680 INFO:teuthology.orchestra.run.smithi105.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-14T00:02:39.681 INFO:teuthology.orchestra.run.smithi105.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-14T00:02:39.681 INFO:teuthology.orchestra.run.smithi105.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-14T00:02:39.681 INFO:teuthology.orchestra.run.smithi105.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-14T00:02:39.681 INFO:teuthology.orchestra.run.smithi105.stdout: python3-watchdog 2023-10-14T00:02:39.681 INFO:teuthology.orchestra.run.smithi105.stdout:Recommended packages: 2023-10-14T00:02:39.682 INFO:teuthology.orchestra.run.smithi105.stdout: btrfs-tools 2023-10-14T00:02:39.771 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2023-10-14T00:02:39.771 INFO:teuthology.orchestra.run.smithi105.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-14T00:02:39.772 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-14T00:02:39.772 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-10-14T00:02:39.772 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2023-10-14T00:02:39.773 INFO:teuthology.orchestra.run.smithi105.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2023-10-14T00:02:39.773 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-14T00:02:39.774 INFO:teuthology.orchestra.run.smithi105.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-14T00:02:39.774 INFO:teuthology.orchestra.run.smithi105.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-14T00:02:39.774 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-14T00:02:39.774 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2023-10-14T00:02:39.775 INFO:teuthology.orchestra.run.smithi105.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-14T00:02:39.775 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-14T00:02:39.775 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-14T00:02:39.776 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-14T00:02:39.776 INFO:teuthology.orchestra.run.smithi105.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-14T00:02:39.776 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-14T00:02:39.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-14T00:02:39.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-14T00:02:39.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-14T00:02:39.778 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-14T00:02:39.778 INFO:teuthology.orchestra.run.smithi105.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-14T00:02:39.778 INFO:teuthology.orchestra.run.smithi105.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-14T00:02:39.778 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-14T00:02:39.779 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-14T00:02:39.779 INFO:teuthology.orchestra.run.smithi105.stdout: socat xmlstarlet 2023-10-14T00:02:39.834 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 109 newly installed, 0 to remove and 256 not upgraded. 2023-10-14T00:02:39.835 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 184 MB of archives. 2023-10-14T00:02:39.835 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 824 MB of additional disk space will be used. 2023-10-14T00:02:39.836 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-14T00:02:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-14T00:02:39.914 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6724-g01fff6a7-1jammy [3,450 kB] 2023-10-14T00:02:39.943 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-14T00:02:39.949 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-14T00:02:39.965 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-14T00:02:40.023 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-14T00:02:40.029 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-14T00:02:40.042 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-14T00:02:40.046 INFO:teuthology.orchestra.run.smithi105.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-14T00:02:40.047 INFO:teuthology.orchestra.run.smithi105.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-14T00:02:40.047 INFO:teuthology.orchestra.run.smithi105.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-14T00:02:40.056 INFO:teuthology.orchestra.run.smithi105.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-14T00:02:40.056 INFO:teuthology.orchestra.run.smithi105.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-14T00:02:40.057 INFO:teuthology.orchestra.run.smithi105.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-14T00:02:40.058 INFO:teuthology.orchestra.run.smithi105.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-14T00:02:40.067 INFO:teuthology.orchestra.run.smithi105.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-14T00:02:40.068 INFO:teuthology.orchestra.run.smithi105.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-14T00:02:40.068 INFO:teuthology.orchestra.run.smithi105.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-14T00:02:40.087 INFO:teuthology.orchestra.run.smithi105.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-14T00:02:40.088 INFO:teuthology.orchestra.run.smithi105.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-14T00:02:40.088 INFO:teuthology.orchestra.run.smithi105.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-14T00:02:40.091 INFO:teuthology.orchestra.run.smithi105.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-14T00:02:40.091 INFO:teuthology.orchestra.run.smithi105.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-14T00:02:40.092 INFO:teuthology.orchestra.run.smithi105.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-14T00:02:40.092 INFO:teuthology.orchestra.run.smithi105.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-14T00:02:40.107 INFO:teuthology.orchestra.run.smithi105.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-14T00:02:40.107 INFO:teuthology.orchestra.run.smithi105.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-14T00:02:40.108 INFO:teuthology.orchestra.run.smithi105.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-14T00:02:40.126 INFO:teuthology.orchestra.run.smithi105.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-14T00:02:40.127 INFO:teuthology.orchestra.run.smithi105.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-14T00:02:40.127 INFO:teuthology.orchestra.run.smithi105.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-14T00:02:40.133 INFO:teuthology.orchestra.run.smithi105.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-14T00:02:40.133 INFO:teuthology.orchestra.run.smithi105.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-14T00:02:40.134 INFO:teuthology.orchestra.run.smithi105.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-14T00:02:40.135 INFO:teuthology.orchestra.run.smithi105.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-14T00:02:40.145 INFO:teuthology.orchestra.run.smithi105.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-14T00:02:40.147 INFO:teuthology.orchestra.run.smithi105.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-14T00:02:40.148 INFO:teuthology.orchestra.run.smithi105.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-14T00:02:40.165 INFO:teuthology.orchestra.run.smithi105.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-14T00:02:40.166 INFO:teuthology.orchestra.run.smithi105.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-14T00:02:40.169 INFO:teuthology.orchestra.run.smithi105.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-14T00:02:40.169 INFO:teuthology.orchestra.run.smithi105.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-14T00:02:40.171 INFO:teuthology.orchestra.run.smithi105.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6724-g01fff6a7-1jammy [3,130 kB] 2023-10-14T00:02:40.172 INFO:teuthology.orchestra.run.smithi105.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-14T00:02:40.228 INFO:teuthology.orchestra.run.smithi105.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-14T00:02:40.229 INFO:teuthology.orchestra.run.smithi105.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-14T00:02:40.230 INFO:teuthology.orchestra.run.smithi105.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-14T00:02:40.230 INFO:teuthology.orchestra.run.smithi105.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-14T00:02:40.231 INFO:teuthology.orchestra.run.smithi105.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-14T00:02:40.231 INFO:teuthology.orchestra.run.smithi105.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-14T00:02:40.257 INFO:teuthology.orchestra.run.smithi105.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-14T00:02:40.259 INFO:teuthology.orchestra.run.smithi105.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-14T00:02:40.260 INFO:teuthology.orchestra.run.smithi105.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-14T00:02:40.282 INFO:teuthology.orchestra.run.smithi105.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-14T00:02:40.283 INFO:teuthology.orchestra.run.smithi105.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-14T00:02:40.283 INFO:teuthology.orchestra.run.smithi105.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-14T00:02:40.286 INFO:teuthology.orchestra.run.smithi105.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6724-g01fff6a7-1jammy [688 kB] 2023-10-14T00:02:40.293 INFO:teuthology.orchestra.run.smithi105.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-14T00:02:40.294 INFO:teuthology.orchestra.run.smithi105.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-14T00:02:40.294 INFO:teuthology.orchestra.run.smithi105.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-14T00:02:40.295 INFO:teuthology.orchestra.run.smithi105.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-14T00:02:40.295 INFO:teuthology.orchestra.run.smithi105.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-14T00:02:40.298 INFO:teuthology.orchestra.run.smithi105.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-14T00:02:40.309 INFO:teuthology.orchestra.run.smithi105.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6724-g01fff6a7-1jammy [356 kB] 2023-10-14T00:02:40.313 INFO:teuthology.orchestra.run.smithi105.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-14T00:02:40.314 INFO:teuthology.orchestra.run.smithi105.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-14T00:02:40.315 INFO:teuthology.orchestra.run.smithi105.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-14T00:02:40.318 INFO:teuthology.orchestra.run.smithi105.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-14T00:02:40.323 INFO:teuthology.orchestra.run.smithi105.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6724-g01fff6a7-1jammy [32.8 kB] 2023-10-14T00:02:40.324 INFO:teuthology.orchestra.run.smithi105.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6724-g01fff6a7-1jammy [180 kB] 2023-10-14T00:02:40.328 INFO:teuthology.orchestra.run.smithi105.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6724-g01fff6a7-1jammy [62.4 kB] 2023-10-14T00:02:40.330 INFO:teuthology.orchestra.run.smithi105.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6724-g01fff6a7-1jammy [330 kB] 2023-10-14T00:02:40.332 INFO:teuthology.orchestra.run.smithi105.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-14T00:02:40.338 INFO:teuthology.orchestra.run.smithi105.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-14T00:02:40.341 INFO:teuthology.orchestra.run.smithi105.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-14T00:02:40.341 INFO:teuthology.orchestra.run.smithi105.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-14T00:02:40.342 INFO:teuthology.orchestra.run.smithi105.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-14T00:02:40.342 INFO:teuthology.orchestra.run.smithi105.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-14T00:02:40.343 INFO:teuthology.orchestra.run.smithi105.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6724-g01fff6a7-1jammy [6,222 kB] 2023-10-14T00:02:40.352 INFO:teuthology.orchestra.run.smithi105.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-14T00:02:40.353 INFO:teuthology.orchestra.run.smithi105.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-14T00:02:40.360 INFO:teuthology.orchestra.run.smithi105.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-14T00:02:40.361 INFO:teuthology.orchestra.run.smithi105.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-14T00:02:40.372 INFO:teuthology.orchestra.run.smithi105.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-14T00:02:40.374 INFO:teuthology.orchestra.run.smithi105.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-14T00:02:40.375 INFO:teuthology.orchestra.run.smithi105.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-14T00:02:40.544 INFO:teuthology.orchestra.run.smithi105.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6724-g01fff6a7-1jammy [112 kB] 2023-10-14T00:02:40.546 INFO:teuthology.orchestra.run.smithi105.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6724-g01fff6a7-1jammy [439 kB] 2023-10-14T00:02:40.560 INFO:teuthology.orchestra.run.smithi105.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6724-g01fff6a7-1jammy [22.6 MB] 2023-10-14T00:02:41.394 INFO:teuthology.orchestra.run.smithi105.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6724-g01fff6a7-1jammy [6,853 kB] 2023-10-14T00:02:41.528 INFO:teuthology.orchestra.run.smithi105.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6724-g01fff6a7-1jammy [238 kB] 2023-10-14T00:02:41.534 INFO:teuthology.orchestra.run.smithi105.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6724-g01fff6a7-1jammy [128 kB] 2023-10-14T00:02:41.538 INFO:teuthology.orchestra.run.smithi105.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6724-g01fff6a7-1jammy [1,496 kB] 2023-10-14T00:02:41.663 INFO:teuthology.orchestra.run.smithi105.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6724-g01fff6a7-1jammy [9,863 kB] 2023-10-14T00:02:41.922 INFO:teuthology.orchestra.run.smithi105.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6724-g01fff6a7-1jammy [31.7 MB] 2023-10-14T00:02:43.067 INFO:teuthology.orchestra.run.smithi105.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6724-g01fff6a7-1jammy [14.0 kB] 2023-10-14T00:02:43.068 INFO:teuthology.orchestra.run.smithi105.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6724-g01fff6a7-1jammy [839 kB] 2023-10-14T00:02:43.086 INFO:teuthology.orchestra.run.smithi105.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6724-g01fff6a7-1jammy [2,146 kB] 2023-10-14T00:02:43.121 INFO:teuthology.orchestra.run.smithi105.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6724-g01fff6a7-1jammy [283 kB] 2023-10-14T00:02:43.125 INFO:teuthology.orchestra.run.smithi105.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6724-g01fff6a7-1jammy [138 kB] 2023-10-14T00:02:43.127 INFO:teuthology.orchestra.run.smithi105.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6724-g01fff6a7-1jammy [1,906 kB] 2023-10-14T00:02:43.149 INFO:teuthology.orchestra.run.smithi105.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6724-g01fff6a7-1jammy [8,625 kB] 2023-10-14T00:02:43.419 INFO:teuthology.orchestra.run.smithi105.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6724-g01fff6a7-1jammy [14.2 kB] 2023-10-14T00:02:43.420 INFO:teuthology.orchestra.run.smithi105.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6724-g01fff6a7-1jammy [47.8 MB] 2023-10-14T00:02:45.248 INFO:teuthology.orchestra.run.smithi105.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6724-g01fff6a7-1jammy [120 kB] 2023-10-14T00:02:45.320 INFO:teuthology.orchestra.run.smithi105.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6724-g01fff6a7-1jammy [38.3 kB] 2023-10-14T00:02:45.321 INFO:teuthology.orchestra.run.smithi105.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6724-g01fff6a7-1jammy [12.3 MB] 2023-10-14T00:02:45.611 INFO:teuthology.orchestra.run.smithi105.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6724-g01fff6a7-1jammy [91.4 kB] 2023-10-14T00:02:46.162 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 184 MB in 6s (31.5 MB/s) 2023-10-14T00:02:46.282 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-14T00:02:46.378 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2023-10-14T00:02:46.383 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-14T00:02:46.472 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-14T00:02:46.833 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-14T00:02:46.859 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-14T00:02:46.900 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-14T00:02:47.237 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-14T00:02:47.263 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-14T00:02:47.303 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-14T00:02:47.615 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-14T00:02:47.641 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-14T00:02:47.681 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-14T00:02:48.126 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-14T00:02:48.153 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-14T00:02:48.201 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:02:48.580 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-14T00:02:48.606 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-14T00:02:48.646 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:02:48.966 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-14T00:02:48.993 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-14T00:02:49.033 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:02:49.394 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-14T00:02:49.420 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-14T00:02:49.461 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-14T00:02:49.788 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librados2. 2023-10-14T00:02:49.815 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../008-librados2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:49.855 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librados2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:50.291 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librbd1. 2023-10-14T00:02:50.318 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../009-librbd1_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:50.350 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librbd1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:50.769 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libcephfs2. 2023-10-14T00:02:50.796 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../010-libcephfs2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:50.836 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libcephfs2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:51.080 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rados. 2023-10-14T00:02:51.095 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../011-python3-rados_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:51.130 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rados (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:51.449 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-14T00:02:51.475 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:02:51.516 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-ceph-argparse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:51.777 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cephfs. 2023-10-14T00:02:51.803 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../013-python3-cephfs_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:51.844 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cephfs (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:52.163 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-14T00:02:52.190 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../014-python3-ceph-common_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:02:52.230 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:52.508 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-14T00:02:52.534 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-14T00:02:52.566 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-14T00:02:52.860 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-prettytable. 2023-10-14T00:02:52.886 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2023-10-14T00:02:52.927 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-14T00:02:53.188 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rbd. 2023-10-14T00:02:53.214 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../017-python3-rbd_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:53.254 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rbd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:53.618 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-14T00:02:53.644 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-14T00:02:53.691 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-14T00:02:53.985 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librgw2. 2023-10-14T00:02:54.011 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../019-librgw2_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:54.043 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librgw2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:54.563 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rgw. 2023-10-14T00:02:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:54.621 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:54.974 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-14T00:02:55.000 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-14T00:02:55.040 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-14T00:02:55.335 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libradosstriper1. 2023-10-14T00:02:55.361 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:55.394 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libradosstriper1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:55.788 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-common. 2023-10-14T00:02:55.814 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../023-ceph-common_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:55.855 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:57.111 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-base. 2023-10-14T00:02:57.137 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../024-ceph-base_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:02:57.187 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-base (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:02:57.898 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-14T00:02:57.925 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-14T00:02:57.965 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-14T00:02:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cheroot. 2023-10-14T00:02:58.216 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-14T00:02:58.251 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-14T00:02:58.571 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-14T00:02:58.597 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-14T00:02:58.637 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-14T00:02:59.265 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-14T00:02:59.292 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-14T00:02:59.332 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-14T00:02:59.610 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-14T00:02:59.636 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-14T00:02:59.676 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-14T00:02:59.913 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-tempora. 2023-10-14T00:02:59.939 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2023-10-14T00:02:59.979 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-14T00:03:00.291 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-portend. 2023-10-14T00:03:00.317 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2023-10-14T00:03:00.357 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-14T00:03:00.660 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-14T00:03:00.686 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-14T00:03:00.727 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-14T00:03:00.988 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-14T00:03:01.003 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-14T00:03:01.063 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-14T00:03:01.417 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-natsort. 2023-10-14T00:03:01.443 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2023-10-14T00:03:01.483 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-14T00:03:01.794 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-logutils. 2023-10-14T00:03:01.821 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2023-10-14T00:03:01.861 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-14T00:03:02.164 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-14T00:03:02.190 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-14T00:03:02.230 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-14T00:03:02.558 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-mako. 2023-10-14T00:03:02.575 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-14T00:03:02.617 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-14T00:03:02.987 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-14T00:03:03.013 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-14T00:03:03.053 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-14T00:03:03.373 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-14T00:03:03.399 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-14T00:03:03.439 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-14T00:03:03.759 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-webob. 2023-10-14T00:03:03.786 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-14T00:03:03.834 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-14T00:03:04.254 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-waitress. 2023-10-14T00:03:04.280 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-14T00:03:04.332 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-14T00:03:04.656 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-tempita. 2023-10-14T00:03:04.683 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-14T00:03:04.723 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-14T00:03:05.026 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-paste. 2023-10-14T00:03:05.052 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-14T00:03:05.092 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-14T00:03:05.437 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-14T00:03:05.454 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-14T00:03:05.495 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-14T00:03:05.731 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-14T00:03:05.747 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-14T00:03:05.781 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-14T00:03:06.085 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-webtest. 2023-10-14T00:03:06.112 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2023-10-14T00:03:06.151 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-14T00:03:06.462 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pecan. 2023-10-14T00:03:06.489 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-14T00:03:06.529 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-14T00:03:06.892 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-certifi. 2023-10-14T00:03:06.919 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2023-10-14T00:03:06.959 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-14T00:03:07.287 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-idna. 2023-10-14T00:03:07.302 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../049-python3-idna_3.3-1_all.deb ... 2023-10-14T00:03:07.336 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-14T00:03:07.590 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-urllib3. 2023-10-14T00:03:07.616 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-14T00:03:07.648 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-14T00:03:07.951 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-requests. 2023-10-14T00:03:07.978 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-14T00:03:08.018 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-14T00:03:08.337 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-14T00:03:08.364 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-14T00:03:08.404 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-14T00:03:08.682 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-14T00:03:08.694 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:08.731 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-14T00:03:09.113 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:09.153 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:09.639 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-openssl. 2023-10-14T00:03:09.666 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../055-python3-openssl_21.0.0-1_all.deb ... 2023-10-14T00:03:09.706 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-14T00:03:10.051 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr. 2023-10-14T00:03:10.077 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../056-ceph-mgr_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:10.117 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:10.521 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mon. 2023-10-14T00:03:10.547 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../057-ceph-mon_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:10.587 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mon (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:11.499 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-osd. 2023-10-14T00:03:11.525 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../058-ceph-osd_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:11.573 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-osd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:14.086 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph. 2023-10-14T00:03:14.113 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../059-ceph_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:14.152 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:14.447 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-fuse. 2023-10-14T00:03:14.474 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../060-ceph-fuse_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:14.513 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:14.834 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mds. 2023-10-14T00:03:14.860 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../061-ceph-mds_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:14.892 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mds (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:15.270 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package cephadm. 2023-10-14T00:03:15.296 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../062-cephadm_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:15.336 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:15.681 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-14T00:03:15.708 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../063-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-14T00:03:15.747 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-14T00:03:16.075 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-14T00:03:16.091 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../064-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-14T00:03:16.134 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-14T00:03:17.122 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-babel. 2023-10-14T00:03:17.149 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../065-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-14T00:03:17.189 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-14T00:03:17.525 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jinja2. 2023-10-14T00:03:17.552 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../066-python3-jinja2_3.0.3-1_all.deb ... 2023-10-14T00:03:17.592 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-14T00:03:17.945 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-14T00:03:17.972 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../067-ceph-mgr-cephadm_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:18.011 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:18.357 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-14T00:03:18.372 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../068-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-14T00:03:18.415 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-14T00:03:18.785 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-14T00:03:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../069-python3-repoze.lru_0.7-2_all.deb ... 2023-10-14T00:03:18.851 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-14T00:03:19.154 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-routes. 2023-10-14T00:03:19.181 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../070-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-14T00:03:19.237 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-14T00:03:19.707 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-14T00:03:19.734 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../071-ceph-mgr-dashboard_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:19.774 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:20.186 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-14T00:03:20.202 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../072-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-14T00:03:20.244 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-14T00:03:20.740 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-joblib. 2023-10-14T00:03:20.767 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../073-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-14T00:03:20.806 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-14T00:03:21.160 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-14T00:03:21.187 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../074-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-14T00:03:21.226 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-14T00:03:21.529 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-sklearn. 2023-10-14T00:03:21.556 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../075-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-14T00:03:21.595 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-14T00:03:22.338 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-14T00:03:22.366 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../076-ceph-mgr-diskprediction-local_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:22.404 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:23.385 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cachetools. 2023-10-14T00:03:23.410 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../077-python3-cachetools_5.0.0-1_all.deb ... 2023-10-14T00:03:23.449 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-14T00:03:23.745 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-14T00:03:23.773 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../078-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-14T00:03:23.812 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-14T00:03:24.182 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-14T00:03:24.209 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../079-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-14T00:03:24.248 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-14T00:03:24.559 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rsa. 2023-10-14T00:03:24.587 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../080-python3-rsa_4.8-1_all.deb ... 2023-10-14T00:03:24.626 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-14T00:03:24.938 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-google-auth. 2023-10-14T00:03:24.965 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../081-python3-google-auth_1.5.1-3_all.deb ... 2023-10-14T00:03:25.004 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-14T00:03:25.324 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-14T00:03:25.352 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../082-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-14T00:03:25.390 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-14T00:03:25.779 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-websocket. 2023-10-14T00:03:25.806 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../083-python3-websocket_1.2.3-1_all.deb ... 2023-10-14T00:03:25.845 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-14T00:03:26.157 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-14T00:03:26.184 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../084-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-14T00:03:26.223 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-14T00:03:26.863 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-14T00:03:26.891 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../085-ceph-mgr-k8sevents_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:26.929 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:27.173 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-14T00:03:27.201 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../086-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-14T00:03:27.231 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-14T00:03:27.576 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-14T00:03:27.604 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../087-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-14T00:03:27.643 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-14T00:03:27.907 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package jq. 2023-10-14T00:03:27.932 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../088-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-14T00:03:27.971 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-14T00:03:28.224 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package socat. 2023-10-14T00:03:28.251 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../089-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-14T00:03:28.287 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-14T00:03:28.576 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package xmlstarlet. 2023-10-14T00:03:28.604 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../090-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-14T00:03:28.642 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-14T00:03:28.997 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-test. 2023-10-14T00:03:29.013 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../091-ceph-test_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:29.047 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-test (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:31.268 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-volume. 2023-10-14T00:03:31.295 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../092-ceph-volume_18.0.0-6724-g01fff6a7-1jammy_all.deb ... 2023-10-14T00:03:31.334 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-volume (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:31.588 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-14T00:03:31.604 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../093-libcephfs-dev_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:31.646 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libcephfs-dev (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:32.024 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package nvme-cli. 2023-10-14T00:03:32.052 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../094-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-14T00:03:32.091 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-14T00:03:32.463 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-14T00:03:32.491 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../095-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-14T00:03:32.529 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-14T00:03:32.966 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-14T00:03:32.993 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../096-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-14T00:03:33.032 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-14T00:03:33.335 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pastescript. 2023-10-14T00:03:33.363 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../097-python3-pastescript_2.0.2-4_all.deb ... 2023-10-14T00:03:33.402 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-14T00:03:33.713 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pluggy. 2023-10-14T00:03:33.741 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../098-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-14T00:03:33.771 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-14T00:03:34.058 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-psutil. 2023-10-14T00:03:34.086 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../099-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-14T00:03:34.124 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-14T00:03:34.453 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-py. 2023-10-14T00:03:34.480 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../100-python3-py_1.10.0-1_all.deb ... 2023-10-14T00:03:34.519 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-14T00:03:34.797 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pygments. 2023-10-14T00:03:34.825 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../101-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-14T00:03:34.939 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-14T00:03:35.410 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-14T00:03:35.438 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-14T00:03:35.476 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-14T00:03:35.787 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-toml. 2023-10-14T00:03:35.815 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../103-python3-toml_0.10.2-1_all.deb ... 2023-10-14T00:03:35.854 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-14T00:03:36.131 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pytest. 2023-10-14T00:03:36.148 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../104-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-14T00:03:36.189 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-14T00:03:36.526 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-simplejson. 2023-10-14T00:03:36.554 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-14T00:03:36.592 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-14T00:03:36.887 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-14T00:03:36.904 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-14T00:03:36.945 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-14T00:03:37.649 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package radosgw. 2023-10-14T00:03:37.678 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../107-radosgw_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:37.716 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking radosgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:38.477 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package rbd-fuse. 2023-10-14T00:03:38.493 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../108-rbd-fuse_18.0.0-6724-g01fff6a7-1jammy_amd64.deb ... 2023-10-14T00:03:38.535 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking rbd-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:38.839 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-14T00:03:39.090 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-14T00:03:39.216 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-14T00:03:39.462 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-14T00:03:39.803 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-14T00:03:40.301 INFO:teuthology.orchestra.run.smithi105.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-14T00:03:40.384 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:40.987 INFO:teuthology.orchestra.run.smithi105.stdout:Adding system user cephadm....done 2023-10-14T00:03:41.342 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-14T00:03:41.760 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-14T00:03:42.037 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-14T00:03:42.180 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-14T00:03:42.431 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-14T00:03:42.682 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-14T00:03:42.800 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-14T00:03:43.093 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-14T00:03:43.445 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-14T00:03:43.696 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-14T00:03:43.980 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-14T00:03:44.223 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-14T00:03:44.482 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-14T00:03:44.817 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-ceph-argparse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:45.068 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-14T00:03:45.194 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-14T00:03:45.462 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-14T00:03:45.729 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-14T00:03:46.272 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-14T00:03:46.398 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-14T00:03:46.657 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-14T00:03:46.925 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-14T00:03:47.218 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-14T00:03:47.912 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-14T00:03:48.163 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-14T00:03:48.280 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-14T00:03:48.390 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-14T00:03:48.666 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-14T00:03:49.050 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-14T00:03:49.285 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-14T00:03:49.653 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-14T00:03:49.954 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-14T00:03:50.306 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-14T00:03:50.557 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-14T00:03:50.683 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:03:50.993 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-idna (3.3-1) ... 2023-10-14T00:03:51.294 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-14T00:03:52.458 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:03:52.592 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-14T00:03:52.860 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-14T00:03:52.986 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-14T00:03:53.120 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-14T00:03:53.422 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-14T00:03:53.676 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-14T00:03:53.907 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:03:54.026 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-14T00:03:54.126 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-14T00:03:54.419 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-14T00:03:54.678 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-14T00:03:54.938 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-14T00:03:55.197 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-14T00:03:55.440 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-14T00:03:55.699 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-14T00:03:55.825 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-14T00:03:56.101 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-14T00:03:56.219 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-14T00:03:56.520 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-14T00:03:56.822 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-14T00:03:57.106 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-14T00:03:57.272 INFO:teuthology.orchestra.run.smithi105.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-14T00:03:57.524 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-14T00:03:57.625 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-14T00:03:57.994 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-14T00:03:58.261 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-14T00:03:58.521 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-14T00:03:58.647 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-14T00:03:58.973 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-14T00:03:59.274 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-14T00:03:59.542 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-14T00:03:59.668 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-14T00:03:59.944 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-14T00:04:00.237 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-14T00:04:00.496 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-14T00:04:00.881 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-14T00:04:01.166 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-14T00:04:01.751 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-14T00:04:01.877 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librados2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.012 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librgw2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.137 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.263 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libcephfs2 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.389 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libradosstriper1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.592 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-14T00:04:02.869 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librbd1 (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:02.995 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-modules-core (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:03.121 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:03.334 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-14T00:04:03.334 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-14T00:04:03.864 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libcephfs-dev (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:03.965 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rados (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:04.091 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-14T00:04:05.304 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rbd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:05.422 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up rbd-fuse (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:05.531 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:05.640 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cephfs (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:05.766 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-common (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:Adding group ceph....done 2023-10-14T00:04:06.702 INFO:teuthology.orchestra.run.smithi105.stdout:Adding system user ceph....done 2023-10-14T00:04:06.879 INFO:teuthology.orchestra.run.smithi105.stdout:Setting system user ceph properties....done 2023-10-14T00:04:06.893 INFO:teuthology.orchestra.run.smithi105.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-14T00:04:07.034 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-14T00:04:07.403 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-14T00:04:07.917 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-test (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:08.043 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up radosgw (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:08.535 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-14T00:04:08.535 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-14T00:04:09.071 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-base (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:09.426 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-14T00:04:09.922 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mds (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:10.159 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-14T00:04:10.159 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-14T00:04:10.674 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:10.887 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-14T00:04:10.887 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-14T00:04:11.450 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-osd (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:11.740 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-14T00:04:11.741 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-14T00:04:12.252 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:12.378 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:12.537 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mon (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:12.748 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-14T00:04:12.749 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-14T00:04:13.330 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-cephadm (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:13.490 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:13.599 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-dashboard (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:13.802 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-volume (18.0.0-6724-g01fff6a7-1jammy) ... 2023-10-14T00:04:14.270 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-14T00:04:15.279 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-14T00:04:15.726 INFO:teuthology.orchestra.run.smithi105.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-14T00:04:15.729 DEBUG:teuthology.parallel:result is None 2023-10-14T00:04:15.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:04:15.880 DEBUG:teuthology.orchestra.run.smithi105:> dpkg-query -W -f '${Version}' ceph 2023-10-14T00:04:15.899 INFO:teuthology.orchestra.run.smithi105.stdout:18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:04:15.899 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:04:15.900 INFO:teuthology.task.install:The correct ceph version 18.0.0-6724-g01fff6a7-1jammy is installed. 2023-10-14T00:04:15.900 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-14T00:04:15.901 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:15.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-14T00:04:15.951 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-14T00:04:15.952 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:15.952 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2023-10-14T00:04:16.007 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-14T00:04:16.058 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-14T00:04:16.059 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:16.059 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-14T00:04:16.111 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-14T00:04:16.167 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-14T00:04:16.168 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:16.168 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2023-10-14T00:04:16.223 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-14T00:04:16.279 INFO:teuthology.run_tasks:Running task ceph... 2023-10-14T00:04:16.351 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-14T00:04:16.352 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /var/log/ceph 2023-10-14T00:04:16.365 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-14T00:04:16.365 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-14T00:04:16.419 INFO:tasks.ceph:Creating extra log directories... 2023-10-14T00:04:16.419 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-14T00:04:16.511 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-14T00:04:16.512 INFO:tasks.ceph:config {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-14T00:04:16.512 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274', 'branch': 'wip-dis-testing', 'description': 'rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7427274', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'rbd default data pool': 'datapool', 'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-qa', 'sha1': '735f68d7bf91d643813a8420fc2b11441f4bd8c4'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 74, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2449, 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f', 'sleep_before_teardown': 0, 'subset': '49/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-qa', 'suite_path': '/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/idryomov/ceph', 'suite_sha1': '735f68d7bf91d643813a8420fc2b11441f4bd8c4', 'targets': {'smithi105.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYvQckp2HFA+3Xt5famVR0+ItJimBVjM4uj2TjVxZTAVlb8a2GoWX8NXDIwcbxdsG5JcnrW4QgTnV7/jDpR5BU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'client.0': ['sudo ceph osd pool create datapool 4', 'rbd pool init datapool']}}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-13_23:35:38', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-14T00:04:16.513 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-14T00:04:16.524 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2023-10-14T00:04:16.580 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:16.581 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2023-10-14T00:04:16.628 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-14T00:04:16.629 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2023-10-14T00:04:16.676 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-14T00:04:16.677 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:16.677 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 888 Links: 1 2023-10-14T00:04:16.678 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:16.678 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-10-13 23:59:10.205329765 +0000 2023-10-14T00:04:16.678 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-10-13 23:59:09.829333118 +0000 2023-10-14T00:04:16.678 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-10-13 23:59:09.829333118 +0000 2023-10-14T00:04:16.678 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-10-14T00:04:16.679 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-14T00:04:16.730 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-10-14T00:04:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-10-14T00:04:16.731 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000373099 s, 1.4 MB/s 2023-10-14T00:04:16.732 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-14T00:04:16.781 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2023-10-14T00:04:16.832 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-14T00:04:16.833 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:16.833 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 910 Links: 1 2023-10-14T00:04:16.833 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:16.833 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-10-13 23:59:10.805324412 +0000 2023-10-14T00:04:16.833 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-10-13 23:59:10.453327553 +0000 2023-10-14T00:04:16.834 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-10-13 23:59:10.453327553 +0000 2023-10-14T00:04:16.834 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-10-14T00:04:16.834 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-14T00:04:16.886 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-10-14T00:04:16.887 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-10-14T00:04:16.887 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000365905 s, 1.4 MB/s 2023-10-14T00:04:16.888 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-14T00:04:16.937 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2023-10-14T00:04:16.984 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-14T00:04:16.984 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:16.984 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 925 Links: 1 2023-10-14T00:04:16.985 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:16.985 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-10-13 23:59:11.393319167 +0000 2023-10-14T00:04:16.985 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-10-13 23:59:11.025322449 +0000 2023-10-14T00:04:16.985 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-10-13 23:59:11.025322449 +0000 2023-10-14T00:04:16.985 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-10-14T00:04:16.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-14T00:04:17.038 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-10-14T00:04:17.039 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-10-14T00:04:17.039 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000395227 s, 1.3 MB/s 2023-10-14T00:04:17.040 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-14T00:04:17.090 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2023-10-14T00:04:17.136 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-14T00:04:17.137 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T00:04:17.137 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 938 Links: 1 2023-10-14T00:04:17.137 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T00:04:17.137 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-10-13 23:59:11.985313886 +0000 2023-10-14T00:04:17.138 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-10-13 23:59:11.617317168 +0000 2023-10-14T00:04:17.138 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-10-13 23:59:11.617317168 +0000 2023-10-14T00:04:17.138 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-10-14T00:04:17.139 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-14T00:04:17.190 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-10-14T00:04:17.191 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-10-14T00:04:17.191 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000404774 s, 1.3 MB/s 2023-10-14T00:04:17.192 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-14T00:04:17.241 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:17.241 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-14T00:04:17.242 INFO:tasks.ceph:Generating config... 2023-10-14T00:04:17.243 INFO:tasks.ceph:[client] rbd default data pool = datapool 2023-10-14T00:04:17.243 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-14T00:04:17.243 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-14T00:04:17.244 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-14T00:04:17.244 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-14T00:04:17.244 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-14T00:04:17.244 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-14T00:04:17.245 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-14T00:04:17.245 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-14T00:04:17.245 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-14T00:04:17.245 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-14T00:04:17.246 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-14T00:04:17.246 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-14T00:04:17.246 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-14T00:04:17.247 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-14T00:04:17.247 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-14T00:04:17.247 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-14T00:04:17.247 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-14T00:04:17.248 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-14T00:04:17.248 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-14T00:04:17.248 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-14T00:04:17.248 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-14T00:04:17.249 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-14T00:04:17.249 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-14T00:04:17.249 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-14T00:04:17.249 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-14T00:04:17.250 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-14T00:04:17.250 INFO:tasks.ceph:Setting up mon.a... 2023-10-14T00:04:17.250 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-14T00:04:17.314 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.keyring 2023-10-14T00:04:17.329 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-14T00:04:17.406 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T00:04:17.459 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.105')] 2023-10-14T00:04:17.459 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.105', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default data pool': 'datapool', 'rbd default features': 1}, 'mon.a': {}} 2023-10-14T00:04:17.460 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:17.460 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T00:04:17.504 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.105 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:17.582 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:17.583 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: generated fsid 0234826a-19f7-4f4f-8da0-4e6264e0bcb3 2023-10-14T00:04:17.583 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = pacific 2023-10-14T00:04:17.583 INFO:teuthology.orchestra.run.smithi105.stdout:epoch 0 2023-10-14T00:04:17.583 INFO:teuthology.orchestra.run.smithi105.stdout:fsid 0234826a-19f7-4f4f-8da0-4e6264e0bcb3 2023-10-14T00:04:17.584 INFO:teuthology.orchestra.run.smithi105.stdout:last_changed 2023-10-14T00:04:17.583800+0000 2023-10-14T00:04:17.584 INFO:teuthology.orchestra.run.smithi105.stdout:created 2023-10-14T00:04:17.583800+0000 2023-10-14T00:04:17.584 INFO:teuthology.orchestra.run.smithi105.stdout:min_mon_release 16 (pacific) 2023-10-14T00:04:17.584 INFO:teuthology.orchestra.run.smithi105.stdout:election_strategy: 1 2023-10-14T00:04:17.585 INFO:teuthology.orchestra.run.smithi105.stdout:0: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.a 2023-10-14T00:04:17.585 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-14T00:04:17.586 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T00:04:17.632 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0234826a-19f7-4f4f-8da0-4e6264e0bcb3... 2023-10-14T00:04:17.633 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-14T00:04:17.703 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2023-10-14T00:04:17.703 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2023-10-14T00:04:17.703 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-14T00:04:17.703 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2023-10-14T00:04:17.704 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.704 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2023-10-14T00:04:17.704 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.704 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2023-10-14T00:04:17.704 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.705 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2023-10-14T00:04:17.705 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2023-10-14T00:04:17.705 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.705 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2023-10-14T00:04:17.706 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2023-10-14T00:04:17.706 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.706 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-14T00:04:17.706 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2023-10-14T00:04:17.707 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.707 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2023-10-14T00:04:17.707 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-14T00:04:17.708 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.708 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2023-10-14T00:04:17.708 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.708 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2023-10-14T00:04:17.709 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2023-10-14T00:04:17.709 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2023-10-14T00:04:17.709 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2023-10-14T00:04:17.710 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2023-10-14T00:04:17.710 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2023-10-14T00:04:17.710 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2023-10-14T00:04:17.711 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-14T00:04:17.711 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-14T00:04:17.711 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2023-10-14T00:04:17.711 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.712 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-14T00:04:17.712 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.712 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2023-10-14T00:04:17.713 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.713 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2023-10-14T00:04:17.713 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.714 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2023-10-14T00:04:17.714 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2023-10-14T00:04:17.714 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2023-10-14T00:04:17.715 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = 172.21.15.105 2023-10-14T00:04:17.715 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2023-10-14T00:04:17.715 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on pool no app = False 2023-10-14T00:04:17.715 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 5000 2023-10-14T00:04:17.716 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = 0234826a-19f7-4f4f-8da0-4e6264e0bcb3 2023-10-14T00:04:17.716 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.716 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2023-10-14T00:04:17.717 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2023-10-14T00:04:17.717 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.717 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2023-10-14T00:04:17.718 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 600 2023-10-14T00:04:17.718 INFO:teuthology.orchestra.run.smithi105.stdout: osd mclock profile = high_recovery_ops 2023-10-14T00:04:17.718 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.719 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2023-10-14T00:04:17.719 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2023-10-14T00:04:17.719 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.720 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2023-10-14T00:04:17.720 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2023-10-14T00:04:17.720 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2023-10-14T00:04:17.721 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.721 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2023-10-14T00:04:17.721 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2023-10-14T00:04:17.721 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.722 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2023-10-14T00:04:17.722 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.722 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2023-10-14T00:04:17.723 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.723 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2023-10-14T00:04:17.723 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.724 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2023-10-14T00:04:17.724 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2023-10-14T00:04:17.724 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.724 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2023-10-14T00:04:17.725 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2023-10-14T00:04:17.725 INFO:teuthology.orchestra.run.smithi105.stdout: bdev async discard = True 2023-10-14T00:04:17.725 INFO:teuthology.orchestra.run.smithi105.stdout: bdev enable discard = True 2023-10-14T00:04:17.726 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore allocator = bitmap 2023-10-14T00:04:17.726 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore block size = 96636764160 2023-10-14T00:04:17.726 INFO:teuthology.orchestra.run.smithi105.stdout: bluestore fsck on mount = True 2023-10-14T00:04:17.727 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluefs = 1/20 2023-10-14T00:04:17.727 INFO:teuthology.orchestra.run.smithi105.stdout: debug bluestore = 1/20 2023-10-14T00:04:17.727 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2023-10-14T00:04:17.727 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2023-10-14T00:04:17.728 INFO:teuthology.orchestra.run.smithi105.stdout: debug rocksdb = 4/10 2023-10-14T00:04:17.728 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd backfillfull_ratio = 0.85 2023-10-14T00:04:17.728 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd full ratio = 0.9 2023-10-14T00:04:17.729 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd nearfull ratio = 0.8 2023-10-14T00:04:17.729 INFO:teuthology.orchestra.run.smithi105.stdout: osd failsafe full ratio = 0.95 2023-10-14T00:04:17.729 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = bluestore 2023-10-14T00:04:17.730 INFO:teuthology.orchestra.run.smithi105.stdout: osd shutdown pgref assert = True 2023-10-14T00:04:17.730 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.730 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2023-10-14T00:04:17.731 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2023-10-14T00:04:17.731 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2023-10-14T00:04:17.731 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2023-10-14T00:04:17.731 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2023-10-14T00:04:17.732 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2023-10-14T00:04:17.732 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2023-10-14T00:04:17.732 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2023-10-14T00:04:17.733 INFO:teuthology.orchestra.run.smithi105.stdout: debug rbd = 20 2023-10-14T00:04:17.733 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.733 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2023-10-14T00:04:17.734 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2023-10-14T00:04:17.734 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2023-10-14T00:04:17.734 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2023-10-14T00:04:17.735 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2023-10-14T00:04:17.735 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2023-10-14T00:04:17.735 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2023-10-14T00:04:17.735 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2023-10-14T00:04:17.735 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2023-10-14T00:04:17.736 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2023-10-14T00:04:17.736 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2023-10-14T00:04:17.736 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.736 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-14T00:04:17.736 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2023-10-14T00:04:17.737 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2023-10-14T00:04:17.737 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.737 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2023-10-14T00:04:17.737 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-14T00:04:17.737 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-14T00:04:17.738 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.738 INFO:teuthology.orchestra.run.smithi105.stdout: # 1m isn't quite enough 2023-10-14T00:04:17.738 INFO:teuthology.orchestra.run.smithi105.stdout: mon_down_mkfs_grace = 2m 2023-10-14T00:04:17.738 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.738 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_filestore_osds = false 2023-10-14T00:04:17.739 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:17.739 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2023-10-14T00:04:17.739 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2023-10-14T00:04:17.739 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2023-10-14T00:04:17.739 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2023-10-14T00:04:17.740 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-14T00:04:17.740 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-14T00:04:17.740 INFO:teuthology.orchestra.run.smithi105.stdout: rbd default data pool = datapool 2023-10-14T00:04:17.740 INFO:teuthology.orchestra.run.smithi105.stdout: rbd default features = 1 2023-10-14T00:04:17.740 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2023-10-14T00:04:17.741 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-14T00:04:17.742 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-14T00:04:17.831 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-14T00:04:17.832 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:17.832 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-14T00:04:17.838 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:17.838 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-14T00:04:17.884 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T00:04:17.884 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:17.885 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-14T00:04:17.885 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T00:04:17.948 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:17.948 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:17.996 INFO:tasks.ceph:Setting up mon nodes... 2023-10-14T00:04:17.996 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-14T00:04:17.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T00:04:18.080 INFO:teuthology.orchestra.run.smithi105.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T00:04:18.083 INFO:tasks.ceph:Setting up mds nodes... 2023-10-14T00:04:18.083 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-14T00:04:18.084 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-14T00:04:18.160 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-14T00:04:18.171 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-14T00:04:18.171 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-14T00:04:18.172 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:18.223 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:18.223 INFO:tasks.ceph:role: osd.0 2023-10-14T00:04:18.223 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T00:04:18.224 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-14T00:04:18.284 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T00:04:18.284 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T00:04:18.285 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T00:04:18.285 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T00:04:18.285 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T00:04:18.285 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-10-14T00:04:18.286 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T00:04:18.286 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T00:04:18.286 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T00:04:18.286 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T00:04:18.771 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-10-14T00:04:18.773 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-10-14T00:04:18.773 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:18.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:18.942 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: /sbin/restorecon: command not found 2023-10-14T00:04:18.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:04:18.944 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:19.000 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:19.001 INFO:tasks.ceph:role: osd.1 2023-10-14T00:04:19.001 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T00:04:19.001 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-14T00:04:19.063 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T00:04:19.063 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T00:04:19.064 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T00:04:19.064 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T00:04:19.064 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T00:04:19.064 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-10-14T00:04:19.065 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T00:04:19.065 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T00:04:19.065 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T00:04:19.065 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T00:04:19.502 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-10-14T00:04:19.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-10-14T00:04:19.505 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:19.601 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:04:19.612 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: /sbin/restorecon: command not found 2023-10-14T00:04:19.612 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:19.666 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-14T00:04:19.666 INFO:tasks.ceph:role: osd.2 2023-10-14T00:04:19.666 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T00:04:19.667 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-14T00:04:19.727 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T00:04:19.728 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T00:04:19.728 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T00:04:19.728 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-14T00:04:19.729 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T00:04:19.729 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-10-14T00:04:19.729 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T00:04:19.729 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-14T00:04:19.729 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T00:04:19.730 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T00:04:20.178 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-10-14T00:04:20.180 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-10-14T00:04:20.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:20.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:04:20.287 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: /sbin/restorecon: command not found 2023-10-14T00:04:20.287 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:20.397 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:20.396+0000 7f2523ae4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-14T00:04:20.398 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:20.396+0000 7f2523ae4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-14T00:04:20.398 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:20.396+0000 7f2523ae4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T00:04:20.399 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:20.396+0000 7f2523ae4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T00:04:20.399 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:20.396+0000 7f2523ae4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T00:04:20.406 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:20.404+0000 7f2523ae4640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-14T00:04:23.433 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-14T00:04:23.448 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:23.534 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:23.532+0000 7f6978d6e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-14T00:04:23.535 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:23.532+0000 7f6978d6e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-14T00:04:23.535 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:23.532+0000 7f6978d6e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T00:04:23.536 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:23.532+0000 7f6978d6e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T00:04:23.536 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:23.532+0000 7f6978d6e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T00:04:23.537 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:23.536+0000 7f6978d6e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-14T00:04:26.568 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-14T00:04:26.627 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:26.710 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:26.708+0000 7fdb4a1e8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-14T00:04:26.711 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:26.708+0000 7fdb4a1e8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-14T00:04:26.711 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:26.708+0000 7fdb4a1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T00:04:26.712 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:26.708+0000 7fdb4a1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T00:04:26.712 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:26.708+0000 7fdb4a1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T00:04:26.712 INFO:teuthology.orchestra.run.smithi105.stderr:2023-10-14T00:04:26.712+0000 7fdb4a1e8640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-14T00:04:29.727 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-14T00:04:29.788 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-14T00:04:29.788 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:29.789 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-14T00:04:29.842 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:29.843 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-14T00:04:29.895 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:29.895 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-14T00:04:29.951 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:29.951 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-14T00:04:30.012 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:30.012 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-14T00:04:30.063 INFO:tasks.ceph:Adding keys to all mons... 2023-10-14T00:04:30.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-14T00:04:30.118 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.x] 2023-10-14T00:04:30.119 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAC2yllpcviBBAATieCCR60f1NXDl4oJ0ky5A== 2023-10-14T00:04:30.119 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.0] 2023-10-14T00:04:30.119 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAE2yllcK69FxAAvxied8MzzkBrOfKGZJ9vGg== 2023-10-14T00:04:30.119 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.1] 2023-10-14T00:04:30.119 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAH2yllkHvzHxAAFc0kxvd0MpiTu3TFyZL9Tg== 2023-10-14T00:04:30.120 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.2] 2023-10-14T00:04:30.120 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAK2yllRDJuKhAAj37gq65/cMrzJ1ysFH0xUQ== 2023-10-14T00:04:30.120 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2023-10-14T00:04:30.120 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQAC2yllqq6hCRAA2b+lTMliK5+PGbnmXS26Mg== 2023-10-14T00:04:30.121 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T00:04:30.212 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T00:04:30.254 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T00:04:30.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T00:04:30.422 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T00:04:30.508 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-14T00:04:30.508 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-14T00:04:30.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T00:04:30.613 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-14T00:04:30.623 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-14T00:04:30.672 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-14T00:04:30.673 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-14T00:04:30.673 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-14T00:04:30.717 INFO:tasks.ceph.mon.a:Started 2023-10-14T00:04:30.717 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-14T00:04:30.718 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-14T00:04:30.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-14T00:04:30.720 INFO:tasks.ceph.mgr.x:Started 2023-10-14T00:04:30.720 INFO:tasks.ceph:Setting crush tunables to default 2023-10-14T00:04:30.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd crush tunables default 2023-10-14T00:04:30.965 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2023-10-14T00:04:30.982 INFO:tasks.ceph:check_enable_crimson: False 2023-10-14T00:04:30.983 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-14T00:04:30.983 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:30.983 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-14T00:04:30.995 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:30.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-14T00:04:31.055 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:31.055 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-14T00:04:31.108 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:31.108+0000 7f942b74d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T00:04:31.110 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 16ee973b-0431-49dd-8db9-d15a5f4d04fc 0 2023-10-14T00:04:31.352 INFO:teuthology.orchestra.run.smithi105.stdout:0 2023-10-14T00:04:31.370 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 3b8038c1-9263-4c09-8d87-77b387bb5668 1 2023-10-14T00:04:31.569 INFO:teuthology.orchestra.run.smithi105.stdout:1 2023-10-14T00:04:31.586 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f 2 2023-10-14T00:04:31.672 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:31.672+0000 7f942b74d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T00:04:31.781 INFO:teuthology.orchestra.run.smithi105.stdout:2 2023-10-14T00:04:31.799 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-14T00:04:31.799 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-14T00:04:31.801 INFO:tasks.ceph.osd.0:Started 2023-10-14T00:04:31.802 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-14T00:04:31.802 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-14T00:04:31.804 INFO:tasks.ceph.osd.1:Started 2023-10-14T00:04:31.804 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-14T00:04:31.805 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-14T00:04:31.807 INFO:tasks.ceph.osd.2:Started 2023-10-14T00:04:31.807 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T00:04:31.894 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:31.892+0000 7f942b74d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T00:04:31.998 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:31.999 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":5,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T00:04:31.781133+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:04:32.015 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-14T00:04:32.016 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-14T00:04:32.318 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T00:04:32.421 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:04:32.420+0000 7f121fd42640 -1 Falling back to public interface 2023-10-14T00:04:32.436 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:04:32.436+0000 7f5bbc011640 -1 Falling back to public interface 2023-10-14T00:04:32.452 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T00:04:32.452+0000 7f34e2304640 -1 Falling back to public interface 2023-10-14T00:04:32.499 INFO:teuthology.misc.health.smithi105.stdout: 2023-10-14T00:04:32.500 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":5,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T00:04:31.781133+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:04:32.515 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-14T00:04:32.553 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.552+0000 7f942b74d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T00:04:32.607 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.608+0000 7f942b74d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T00:04:32.659 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.656+0000 7f942b74d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T00:04:32.735 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.732+0000 7f942b74d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T00:04:32.787 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.784+0000 7f942b74d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T00:04:32.863 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.860+0000 7f942b74d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T00:04:32.914 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:32.912+0000 7f942b74d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T00:04:33.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:33.052+0000 7f942b74d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T00:04:33.111 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:33.108+0000 7f942b74d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T00:04:33.711 INFO:tasks.ceph.mgr.x.smithi105.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-14T00:04:33.712 INFO:tasks.ceph.mgr.x.smithi105.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-14T00:04:33.712 INFO:tasks.ceph.mgr.x.smithi105.stderr: from numpy import show_config as show_numpy_config 2023-10-14T00:04:33.774 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:33.772+0000 7f942b74d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T00:04:33.911 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:33.908+0000 7f942b74d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T00:04:34.063 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:34.060+0000 7f942b74d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T00:04:34.125 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:34.124+0000 7f942b74d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T00:04:34.246 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:34.244+0000 7f942b74d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T00:04:34.835 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:34.832+0000 7f942b74d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T00:04:34.910 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:34.908+0000 7f942b74d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T00:04:35.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:35.080+0000 7f942b74d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T00:04:35.158 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:35.156+0000 7f942b74d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T00:04:35.458 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:35.456+0000 7f942b74d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T00:04:35.532 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:35.532+0000 7f942b74d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T00:04:36.058 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T00:04:36.055+0000 7f34e2304640 -1 osd.2 0 log_to_monitors true 2023-10-14T00:04:36.260 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:04:36.259+0000 7f121fd42640 -1 osd.0 0 log_to_monitors true 2023-10-14T00:04:36.350 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:04:36.347+0000 7f5bbc011640 -1 osd.1 0 log_to_monitors true 2023-10-14T00:04:36.568 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:04:36.567+0000 7f93d30c2640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-14T00:04:38.817 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T00:04:39.291 INFO:teuthology.misc.health.smithi105.stdout: 2023-10-14T00:04:39.291 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":7,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T00:04:37.803044+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:04:39.317 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-14T00:04:39.706 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:04:39.703+0000 7f121ccff640 -1 osd.0 0 waiting for initial osdmap 2023-10-14T00:04:39.708 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T00:04:39.707+0000 7f34df2c1640 -1 osd.2 0 waiting for initial osdmap 2023-10-14T00:04:39.712 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:04:39.711+0000 7f5bb7796640 -1 osd.1 0 waiting for initial osdmap 2023-10-14T00:04:39.723 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:04:39.723+0000 7f1213b0e640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T00:04:39.743 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T00:04:39.743+0000 7f34d60d0640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T00:04:39.751 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:04:39.751+0000 7f5bafddd640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T00:04:45.619 DEBUG:teuthology.orchestra.run.smithi105:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T00:04:45.910 INFO:teuthology.misc.health.smithi105.stdout: 2023-10-14T00:04:45.910 INFO:teuthology.misc.health.smithi105.stdout:{"epoch":11,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T00:04:43.860883+0000","last_up_change":"2023-10-14T00:04:40.716111+0000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:04:41.634304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6801","nonce":2607237379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6803","nonce":2607237379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6807","nonce":2607237379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6805","nonce":2607237379}]},"public_addr":"172.21.15.105:6801/2607237379","cluster_addr":"172.21.15.105:6803/2607237379","heartbeat_back_addr":"172.21.15.105:6807/2607237379","heartbeat_front_addr":"172.21.15.105:6805/2607237379","state":["exists","up"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6809","nonce":2843110786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6811","nonce":2843110786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6815","nonce":2843110786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6813","nonce":2843110786}]},"public_addr":"172.21.15.105:6809/2843110786","cluster_addr":"172.21.15.105:6811/2843110786","heartbeat_back_addr":"172.21.15.105:6815/2843110786","heartbeat_front_addr":"172.21.15.105:6813/2843110786","state":["exists","up"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6817","nonce":4190342661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6819","nonce":4190342661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6823","nonce":4190342661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6821","nonce":4190342661}]},"public_addr":"172.21.15.105:6817/4190342661","cluster_addr":"172.21.15.105:6819/4190342661","heartbeat_back_addr":"172.21.15.105:6823/4190342661","heartbeat_front_addr":"172.21.15.105:6821/4190342661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.250185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.382036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.014142+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:04:45.927 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-14T00:04:45.927 INFO:tasks.ceph:Creating RBD pool 2023-10-14T00:04:45.928 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-14T00:04:46.871 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'rbd' created 2023-10-14T00:04:46.900 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-14T00:04:47.873 INFO:teuthology.orchestra.run.smithi105.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-14T00:04:47.901 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-14T00:04:47.902 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-14T00:04:47.902 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-14T00:04:47.903 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-14T00:04:47.903 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-14T00:04:48.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:48.215 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3561963830},{"type":"v1","addr":"172.21.15.105:6825","nonce":3561963830}]},"active_addr":"172.21.15.105:6825/3561963830","active_change":"2023-10-14T00:04:35.554375+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":2331715049}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":952497813}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":3591507769}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":956713934}]}]} 2023-10-14T00:04:48.217 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-14T00:04:48.218 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-14T00:04:48.218 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T00:04:48.502 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:48.503 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":13,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T00:04:47.872495+0000","last_up_change":"2023-10-14T00:04:40.716111+0000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:04:41.634304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-14T00:04:46.262162+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6801","nonce":2607237379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6803","nonce":2607237379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6807","nonce":2607237379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6805","nonce":2607237379}]},"public_addr":"172.21.15.105:6801/2607237379","cluster_addr":"172.21.15.105:6803/2607237379","heartbeat_back_addr":"172.21.15.105:6807/2607237379","heartbeat_front_addr":"172.21.15.105:6805/2607237379","state":["exists","up"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6809","nonce":2843110786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6811","nonce":2843110786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6815","nonce":2843110786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6813","nonce":2843110786}]},"public_addr":"172.21.15.105:6809/2843110786","cluster_addr":"172.21.15.105:6811/2843110786","heartbeat_back_addr":"172.21.15.105:6815/2843110786","heartbeat_front_addr":"172.21.15.105:6813/2843110786","state":["exists","up"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6817","nonce":4190342661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6819","nonce":4190342661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6823","nonce":4190342661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6821","nonce":4190342661}]},"public_addr":"172.21.15.105:6817/4190342661","cluster_addr":"172.21.15.105:6819/4190342661","heartbeat_back_addr":"172.21.15.105:6823/4190342661","heartbeat_front_addr":"172.21.15.105:6821/4190342661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.250185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.382036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.014142+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:04:48.519 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-14T00:04:48.520 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T00:04:48.798 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:48.799 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":13,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T00:04:47.872495+0000","last_up_change":"2023-10-14T00:04:40.716111+0000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:04:41.634304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-14T00:04:46.262162+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6801","nonce":2607237379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6803","nonce":2607237379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6807","nonce":2607237379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6805","nonce":2607237379}]},"public_addr":"172.21.15.105:6801/2607237379","cluster_addr":"172.21.15.105:6803/2607237379","heartbeat_back_addr":"172.21.15.105:6807/2607237379","heartbeat_front_addr":"172.21.15.105:6805/2607237379","state":["exists","up"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6809","nonce":2843110786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6811","nonce":2843110786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6815","nonce":2843110786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6813","nonce":2843110786}]},"public_addr":"172.21.15.105:6809/2843110786","cluster_addr":"172.21.15.105:6811/2843110786","heartbeat_back_addr":"172.21.15.105:6815/2843110786","heartbeat_front_addr":"172.21.15.105:6813/2843110786","state":["exists","up"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6817","nonce":4190342661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6819","nonce":4190342661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6823","nonce":4190342661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6821","nonce":4190342661}]},"public_addr":"172.21.15.105:6817/4190342661","cluster_addr":"172.21.15.105:6819/4190342661","heartbeat_back_addr":"172.21.15.105:6823/4190342661","heartbeat_front_addr":"172.21.15.105:6821/4190342661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.250185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.382036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.014142+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T00:04:48.816 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-14T00:04:48.817 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-14T00:04:48.817 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-14T00:04:48.969 INFO:teuthology.orchestra.run.smithi105.stdout:34359738374 2023-10-14T00:04:48.969 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T00:04:48.970 INFO:teuthology.orchestra.run.smithi105.stdout:34359738374 2023-10-14T00:04:48.971 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T00:04:48.972 INFO:teuthology.orchestra.run.smithi105.stdout:34359738373 2023-10-14T00:04:48.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T00:04:49.279 INFO:teuthology.orchestra.run.smithi105.stdout:34359738373 2023-10-14T00:04:49.296 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.1 2023-10-14T00:04:49.305 INFO:teuthology.orchestra.run.smithi105.stdout:34359738372 2023-10-14T00:04:49.307 INFO:teuthology.orchestra.run.smithi105.stdout:34359738373 2023-10-14T00:04:49.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-14T00:04:49.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738373 for osd.0 2023-10-14T00:04:50.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T00:04:50.325 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T00:04:50.326 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T00:04:50.606 INFO:teuthology.orchestra.run.smithi105.stdout:34359738374 2023-10-14T00:04:50.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2023-10-14T00:04:50.623 DEBUG:teuthology.parallel:result is None 2023-10-14T00:04:50.630 INFO:teuthology.orchestra.run.smithi105.stdout:34359738373 2023-10-14T00:04:50.635 INFO:teuthology.orchestra.run.smithi105.stdout:34359738374 2023-10-14T00:04:50.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-14T00:04:50.649 DEBUG:teuthology.parallel:result is None 2023-10-14T00:04:50.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-14T00:04:50.654 DEBUG:teuthology.parallel:result is None 2023-10-14T00:04:50.654 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T00:04:50.655 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T00:04:50.961 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:50.961 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T00:04:50.977 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-14T00:04:49.562218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.688377"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897235+0000","last_change":"2023-10-14T00:04:47.892979+0000","last_active":"2023-10-14T00:04:47.897235+0000","last_peered":"2023-10-14T00:04:47.897235+0000","last_clean":"2023-10-14T00:04:47.897235+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T00:04:47.897235+0000","last_undegraded":"2023-10-14T00:04:47.897235+0000","last_fullsized":"2023-10-14T00:04:47.897235+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:39:54.245192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897253+0000","last_change":"2023-10-14T00:04:47.892672+0000","last_active":"2023-10-14T00:04:47.897253+0000","last_peered":"2023-10-14T00:04:47.897253+0000","last_clean":"2023-10-14T00:04:47.897253+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T00:04:47.897253+0000","last_undegraded":"2023-10-14T00:04:47.897253+0000","last_fullsized":"2023-10-14T00:04:47.897253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:19:02.693600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897303+0000","last_change":"2023-10-14T00:04:47.893288+0000","last_active":"2023-10-14T00:04:47.897303+0000","last_peered":"2023-10-14T00:04:47.897303+0000","last_clean":"2023-10-14T00:04:47.897303+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T00:04:47.897303+0000","last_undegraded":"2023-10-14T00:04:47.897303+0000","last_fullsized":"2023-10-14T00:04:47.897303+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:46:25.230904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897146+0000","last_change":"2023-10-14T00:04:47.892782+0000","last_active":"2023-10-14T00:04:47.897146+0000","last_peered":"2023-10-14T00:04:47.897146+0000","last_clean":"2023-10-14T00:04:47.897146+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T00:04:47.897146+0000","last_undegraded":"2023-10-14T00:04:47.897146+0000","last_fullsized":"2023-10-14T00:04:47.897146+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:34:03.569699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:04:48.897945+0000","last_change":"2023-10-14T00:04:47.890020+0000","last_active":"2023-10-14T00:04:48.897945+0000","last_peered":"2023-10-14T00:04:48.897945+0000","last_clean":"2023-10-14T00:04:48.897945+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T00:04:48.897945+0000","last_undegraded":"2023-10-14T00:04:48.897945+0000","last_fullsized":"2023-10-14T00:04:48.897945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:42:10.640763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.891772+0000","last_change":"2023-10-14T00:04:47.889952+0000","last_active":"2023-10-14T00:04:47.891772+0000","last_peered":"2023-10-14T00:04:47.891772+0000","last_clean":"2023-10-14T00:04:47.891772+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T00:04:47.891772+0000","last_undegraded":"2023-10-14T00:04:47.891772+0000","last_fullsized":"2023-10-14T00:04:47.891772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:11:04.441290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.894615+0000","last_change":"2023-10-14T00:04:47.890489+0000","last_active":"2023-10-14T00:04:47.894615+0000","last_peered":"2023-10-14T00:04:47.894615+0000","last_clean":"2023-10-14T00:04:47.894615+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T00:04:47.894615+0000","last_undegraded":"2023-10-14T00:04:47.894615+0000","last_fullsized":"2023-10-14T00:04:47.894615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:37:24.943928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.892950+0000","last_change":"2023-10-14T00:04:47.888394+0000","last_active":"2023-10-14T00:04:47.892950+0000","last_peered":"2023-10-14T00:04:47.892950+0000","last_clean":"2023-10-14T00:04:47.892950+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T00:04:47.892950+0000","last_undegraded":"2023-10-14T00:04:47.892950+0000","last_fullsized":"2023-10-14T00:04:47.892950+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:16:02.382662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-14T00:04:46.882465+0000","last_change":"2023-10-14T00:04:42.867239+0000","last_active":"2023-10-14T00:04:46.882465+0000","last_peered":"2023-10-14T00:04:46.882465+0000","last_clean":"2023-10-14T00:04:46.882465+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T00:04:46.882465+0000","last_undegraded":"2023-10-14T00:04:46.882465+0000","last_fullsized":"2023-10-14T00:04:46.882465+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_clean_scrub_stamp":"2023-10-14T00:04:41.840847+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:45:11.022601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T00:04:50.978 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T00:04:51.238 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:51.239 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T00:04:51.254 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-14T00:04:49.562218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82564,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032956,"statfs":{"total":289910292480,"available":289825746944,"internally_reserved":0,"allocated":1531904,"data_stored":1274693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4782,"internal_metadata":82963794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.688377"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897235+0000","last_change":"2023-10-14T00:04:47.892979+0000","last_active":"2023-10-14T00:04:47.897235+0000","last_peered":"2023-10-14T00:04:47.897235+0000","last_clean":"2023-10-14T00:04:47.897235+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T00:04:47.897235+0000","last_undegraded":"2023-10-14T00:04:47.897235+0000","last_fullsized":"2023-10-14T00:04:47.897235+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:39:54.245192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897253+0000","last_change":"2023-10-14T00:04:47.892672+0000","last_active":"2023-10-14T00:04:47.897253+0000","last_peered":"2023-10-14T00:04:47.897253+0000","last_clean":"2023-10-14T00:04:47.897253+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T00:04:47.897253+0000","last_undegraded":"2023-10-14T00:04:47.897253+0000","last_fullsized":"2023-10-14T00:04:47.897253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:19:02.693600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897303+0000","last_change":"2023-10-14T00:04:47.893288+0000","last_active":"2023-10-14T00:04:47.897303+0000","last_peered":"2023-10-14T00:04:47.897303+0000","last_clean":"2023-10-14T00:04:47.897303+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T00:04:47.897303+0000","last_undegraded":"2023-10-14T00:04:47.897303+0000","last_fullsized":"2023-10-14T00:04:47.897303+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:46:25.230904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.897146+0000","last_change":"2023-10-14T00:04:47.892782+0000","last_active":"2023-10-14T00:04:47.897146+0000","last_peered":"2023-10-14T00:04:47.897146+0000","last_clean":"2023-10-14T00:04:47.897146+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T00:04:47.897146+0000","last_undegraded":"2023-10-14T00:04:47.897146+0000","last_fullsized":"2023-10-14T00:04:47.897146+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:34:03.569699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-14T00:04:48.897945+0000","last_change":"2023-10-14T00:04:47.890020+0000","last_active":"2023-10-14T00:04:48.897945+0000","last_peered":"2023-10-14T00:04:48.897945+0000","last_clean":"2023-10-14T00:04:48.897945+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T00:04:48.897945+0000","last_undegraded":"2023-10-14T00:04:48.897945+0000","last_fullsized":"2023-10-14T00:04:48.897945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:42:10.640763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.891772+0000","last_change":"2023-10-14T00:04:47.889952+0000","last_active":"2023-10-14T00:04:47.891772+0000","last_peered":"2023-10-14T00:04:47.891772+0000","last_clean":"2023-10-14T00:04:47.891772+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T00:04:47.891772+0000","last_undegraded":"2023-10-14T00:04:47.891772+0000","last_fullsized":"2023-10-14T00:04:47.891772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:11:04.441290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.894615+0000","last_change":"2023-10-14T00:04:47.890489+0000","last_active":"2023-10-14T00:04:47.894615+0000","last_peered":"2023-10-14T00:04:47.894615+0000","last_clean":"2023-10-14T00:04:47.894615+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T00:04:47.894615+0000","last_undegraded":"2023-10-14T00:04:47.894615+0000","last_fullsized":"2023-10-14T00:04:47.894615+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:37:24.943928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-14T00:04:47.892950+0000","last_change":"2023-10-14T00:04:47.888394+0000","last_active":"2023-10-14T00:04:47.892950+0000","last_peered":"2023-10-14T00:04:47.892950+0000","last_clean":"2023-10-14T00:04:47.892950+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T00:04:47.892950+0000","last_undegraded":"2023-10-14T00:04:47.892950+0000","last_fullsized":"2023-10-14T00:04:47.892950+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:16:02.382662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-14T00:04:46.882465+0000","last_change":"2023-10-14T00:04:42.867239+0000","last_active":"2023-10-14T00:04:46.882465+0000","last_peered":"2023-10-14T00:04:46.882465+0000","last_clean":"2023-10-14T00:04:46.882465+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T00:04:46.882465+0000","last_undegraded":"2023-10-14T00:04:46.882465+0000","last_fullsized":"2023-10-14T00:04:46.882465+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_clean_scrub_stamp":"2023-10-14T00:04:41.840847+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:45:11.022601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T00:04:51.255 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T00:04:51.255 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-14T00:04:51.256 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-14T00:04:51.256 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-14T00:04:51.552 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T00:04:51.552 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-14T00:04:51.568 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-14T00:04:51.569 INFO:teuthology.run_tasks:Running task exec... 2023-10-14T00:04:51.738 INFO:teuthology.task.exec:Executing custom commands... 2023-10-14T00:04:51.739 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T00:04:51.739 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create datapool 4' 2023-10-14T00:04:52.909 INFO:teuthology.orchestra.run.smithi105.stderr:pool 'datapool' created 2023-10-14T00:04:52.936 DEBUG:teuthology.orchestra.run.smithi105:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'rbd pool init datapool' 2023-10-14T00:04:55.973 INFO:teuthology.run_tasks:Running task install... 2023-10-14T00:04:55.989 DEBUG:teuthology.task.install:project ceph 2023-10-14T00:04:55.989 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'}} 2023-10-14T00:04:55.989 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': '01fff6a72a328459c1d153e5dc1de6a34e48a82f'} 2023-10-14T00:04:55.990 INFO:teuthology.task.install:Using flavor: default 2023-10-14T00:04:55.995 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-14T00:04:55.995 INFO:teuthology.task.install:extra packages: [] 2023-10-14T00:04:55.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-key list | grep Ceph 2023-10-14T00:04:56.060 INFO:teuthology.orchestra.run.smithi105.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-14T00:04:56.089 INFO:teuthology.orchestra.run.smithi105.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-14T00:04:56.090 INFO:teuthology.orchestra.run.smithi105.stdout:uid [ unknown] Ceph.com (release key) 2023-10-14T00:04:56.091 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-10-14T00:04:56.091 INFO:teuthology.task.install.deb:Installing system (non-project) packages: fio on remote deb x86_64 2023-10-14T00:04:56.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:04:56.236 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default/ 2023-10-14T00:04:56.236 INFO:teuthology.task.install.deb:Package version is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:04:56.615 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:04:56.616 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-14T00:04:56.631 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2023-10-14T00:04:56.935 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-14T00:04:56.935 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-14T00:04:57.006 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-14T00:04:57.088 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-14T00:04:57.137 INFO:teuthology.orchestra.run.smithi105.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy InRelease 2023-10-14T00:04:57.224 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:6 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release 2023-10-14T00:04:57.315 INFO:teuthology.orchestra.run.smithi105.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-dis-testing/01fff6a72a328459c1d153e5dc1de6a34e48a82f/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-14T00:04:58.282 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-14T00:04:58.295 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6724-g01fff6a7-1jammy cephadm=18.0.0-6724-g01fff6a7-1jammy ceph-mds=18.0.0-6724-g01fff6a7-1jammy ceph-mgr=18.0.0-6724-g01fff6a7-1jammy ceph-common=18.0.0-6724-g01fff6a7-1jammy ceph-fuse=18.0.0-6724-g01fff6a7-1jammy ceph-test=18.0.0-6724-g01fff6a7-1jammy ceph-volume=18.0.0-6724-g01fff6a7-1jammy radosgw=18.0.0-6724-g01fff6a7-1jammy python3-rados=18.0.0-6724-g01fff6a7-1jammy python3-rgw=18.0.0-6724-g01fff6a7-1jammy python3-cephfs=18.0.0-6724-g01fff6a7-1jammy python3-rbd=18.0.0-6724-g01fff6a7-1jammy libcephfs2=18.0.0-6724-g01fff6a7-1jammy libcephfs-dev=18.0.0-6724-g01fff6a7-1jammy librados2=18.0.0-6724-g01fff6a7-1jammy librbd1=18.0.0-6724-g01fff6a7-1jammy rbd-fuse=18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:04:58.357 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-14T00:04:58.543 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2023-10-14T00:04:58.543 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2023-10-14T00:04:58.709 INFO:teuthology.orchestra.run.smithi105.stdout:ceph is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.709 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-common is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.709 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-fuse is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.709 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-mds is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.710 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-mgr is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.710 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-test is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.710 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-volume is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.710 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.710 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-dev is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.711 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs2 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.711 INFO:teuthology.orchestra.run.smithi105.stdout:librados2 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.711 INFO:teuthology.orchestra.run.smithi105.stdout:librbd1 is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.711 INFO:teuthology.orchestra.run.smithi105.stdout:python3-cephfs is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.711 INFO:teuthology.orchestra.run.smithi105.stdout:python3-rados is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.712 INFO:teuthology.orchestra.run.smithi105.stdout:python3-rbd is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.712 INFO:teuthology.orchestra.run.smithi105.stdout:python3-rgw is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.712 INFO:teuthology.orchestra.run.smithi105.stdout:radosgw is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.712 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse is already the newest version (18.0.0-6724-g01fff6a7-1jammy). 2023-10-14T00:04:58.712 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2023-10-14T00:04:58.712 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-14T00:04:58.713 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-14T00:04:58.713 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-14T00:04:58.713 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-14T00:04:58.714 INFO:teuthology.orchestra.run.smithi105.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-14T00:04:58.714 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-14T00:04:58.747 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 0 newly installed, 0 to remove and 256 not upgraded. 2023-10-14T00:04:58.748 INFO:teuthology.orchestra.run.smithi105.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-14T00:04:58.750 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install fio 2023-10-14T00:04:58.806 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2023-10-14T00:04:58.991 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2023-10-14T00:04:58.991 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2023-10-14T00:04:59.150 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2023-10-14T00:04:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libiscsi7 libpmemobj1 2023-10-14T00:04:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2-dev libpython2-stdlib libpython2.7 libpython2.7-dev 2023-10-14T00:04:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7-minimal libpython2.7-stdlib linux-image-5.4.0-135-generic 2023-10-14T00:04:59.151 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-5.4.0-135-generic python2 python2-dev python2-minimal 2023-10-14T00:04:59.152 INFO:teuthology.orchestra.run.smithi105.stdout: python2.7 python2.7-dev python2.7-minimal 2023-10-14T00:04:59.152 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-14T00:04:59.152 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2023-10-14T00:04:59.153 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemblk1 2023-10-14T00:04:59.153 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2023-10-14T00:04:59.153 INFO:teuthology.orchestra.run.smithi105.stdout: gnuplot gfio python-scipy 2023-10-14T00:04:59.178 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2023-10-14T00:04:59.180 INFO:teuthology.orchestra.run.smithi105.stdout: fio libpmemblk1 2023-10-14T00:04:59.359 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 2 newly installed, 0 to remove and 256 not upgraded. 2023-10-14T00:04:59.360 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 4,112 kB of archives. 2023-10-14T00:04:59.360 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 7,813 kB of additional disk space will be used. 2023-10-14T00:04:59.360 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libpmemblk1 amd64 1.11.1-3build1 [65.6 kB] 2023-10-14T00:04:59.641 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 fio amd64 3.28-1 [4,047 kB] 2023-10-14T00:05:00.492 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 4,112 kB in 1s (4,279 kB/s) 2023-10-14T00:05:00.649 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libpmemblk1:amd64. 2023-10-14T00:05:00.751 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172814 files and directories currently installed.) 2023-10-14T00:05:00.757 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../libpmemblk1_1.11.1-3build1_amd64.deb ... 2023-10-14T00:05:00.838 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-14T00:05:01.209 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package fio. 2023-10-14T00:05:01.237 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../archives/fio_3.28-1_amd64.deb ... 2023-10-14T00:05:01.275 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking fio (3.28-1) ... 2023-10-14T00:05:01.860 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libpmemblk1:amd64 (1.11.1-3build1) ... 2023-10-14T00:05:01.981 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up fio (3.28-1) ... 2023-10-14T00:05:02.492 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-14T00:05:03.709 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-14T00:05:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-14T00:05:04.173 DEBUG:teuthology.parallel:result is None 2023-10-14T00:05:04.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=01fff6a72a328459c1d153e5dc1de6a34e48a82f 2023-10-14T00:05:04.314 DEBUG:teuthology.orchestra.run.smithi105:> dpkg-query -W -f '${Version}' ceph 2023-10-14T00:05:04.333 INFO:teuthology.orchestra.run.smithi105.stdout:18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:05:04.334 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6724-g01fff6a7-1jammy 2023-10-14T00:05:04.334 INFO:teuthology.task.install:The correct ceph version 18.0.0-6724-g01fff6a7-1jammy is installed. 2023-10-14T00:05:04.335 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-14T00:05:04.335 INFO:teuthology.run_tasks:Running task workunit... 2023-10-14T00:05:04.347 INFO:tasks.workunit:Pulling workunits from ref 735f68d7bf91d643813a8420fc2b11441f4bd8c4 2023-10-14T00:05:04.348 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-14T00:05:04.348 DEBUG:teuthology.orchestra.run.smithi105:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-14T00:05:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T00:05:04.383 INFO:teuthology.orchestra.run.smithi105.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-14T00:05:04.383 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-14T00:05:04.432 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-14T00:05:04.432 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-14T00:05:04.480 INFO:tasks.workunit:timeout=3h 2023-10-14T00:05:04.480 INFO:tasks.workunit:cleanup=True 2023-10-14T00:05:04.481 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/idryomov/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 735f68d7bf91d643813a8420fc2b11441f4bd8c4 2023-10-14T00:05:04.530 INFO:tasks.workunit.client.0.smithi105.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-14T00:05:38.848 INFO:tasks.workunit.client.0.smithi105.stderr:Note: switching to '735f68d7bf91d643813a8420fc2b11441f4bd8c4'. 2023-10-14T00:05:38.848 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-14T00:05:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-14T00:05:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:state without impacting any branches by switching back to a branch. 2023-10-14T00:05:38.849 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-14T00:05:38.850 INFO:tasks.workunit.client.0.smithi105.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-14T00:05:38.850 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.850 INFO:tasks.workunit.client.0.smithi105.stderr: git switch -c 2023-10-14T00:05:38.850 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.850 INFO:tasks.workunit.client.0.smithi105.stderr:Or undo this operation with: 2023-10-14T00:05:38.851 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.851 INFO:tasks.workunit.client.0.smithi105.stderr: git switch - 2023-10-14T00:05:38.851 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.851 INFO:tasks.workunit.client.0.smithi105.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-14T00:05:38.851 INFO:tasks.workunit.client.0.smithi105.stderr: 2023-10-14T00:05:38.852 INFO:tasks.workunit.client.0.smithi105.stderr:HEAD is now at 735f68d7bf9 Merge branch 'wip-62891' of https://github.com/ajarr/ceph into HEAD 2023-10-14T00:05:38.854 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-14T00:05:38.899 INFO:tasks.workunit.client.0.smithi105.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-14T00:05:38.903 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-14T00:05:38.903 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-14T00:05:39.640 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-14T00:05:39.723 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-14T00:05:39.782 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-14T00:05:39.784 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-14T00:05:39.784 INFO:tasks.workunit.client.0.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-14T00:05:39.829 INFO:tasks.workunit.client.0.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-14T00:05:39.832 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-10-14T00:05:39.832 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-14T00:05:39.883 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-14T00:05:39.884 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-14T00:05:39.884 DEBUG:teuthology.orchestra.run.smithi105:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-14T00:05:39.935 INFO:tasks.workunit.client.0.smithi105.stderr:+ POOL=rbd 2023-10-14T00:05:39.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ IMAGE_PREFIX=image 2023-10-14T00:05:39.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ NUM_IMAGES=20 2023-10-14T00:05:39.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ RUN_TIME=3600 2023-10-14T00:05:39.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror pool enable rbd image 2023-10-14T00:05:39.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-14T00:05:40.020 INFO:tasks.workunit.client.0.smithi105.stdout:96607467-3155-4e22-b3d8-183db30e204f 2023-10-14T00:05:40.023 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i = 1 )) 2023-10-14T00:05:40.024 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:40.024 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image1 2023-10-14T00:05:40.072 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-14T00:05:40.883 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:40.896 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-14T00:05:40.958 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:40.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:40.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image2 2023-10-14T00:05:41.006 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-14T00:05:41.888 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:41.901 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-14T00:05:41.960 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:41.960 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:41.960 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image3 2023-10-14T00:05:42.006 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-14T00:05:42.899 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:42.913 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-14T00:05:42.974 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:42.974 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:42.974 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image4 2023-10-14T00:05:43.023 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-14T00:05:43.904 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:43.918 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-14T00:05:43.980 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:43.980 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:43.981 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image5 2023-10-14T00:05:44.030 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-14T00:05:44.917 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:44.930 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-14T00:05:44.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:44.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:44.992 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image6 2023-10-14T00:05:45.038 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-14T00:05:45.917 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:45.928 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-14T00:05:45.989 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:45.989 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:45.989 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image7 2023-10-14T00:05:46.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-14T00:05:46.927 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:46.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-14T00:05:46.997 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:46.997 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:46.998 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image8 2023-10-14T00:05:47.046 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-14T00:05:47.939 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:47.953 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-14T00:05:48.013 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:48.013 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:48.013 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image9 2023-10-14T00:05:48.062 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-14T00:05:48.952 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:48.965 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-14T00:05:49.026 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:49.026 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:49.026 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image10 2023-10-14T00:05:49.074 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-14T00:05:49.955 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:49.967 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-14T00:05:50.029 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:50.029 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:50.030 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image11 2023-10-14T00:05:50.077 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-14T00:05:50.957 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:50.970 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-14T00:05:51.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:51.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:51.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image12 2023-10-14T00:05:51.080 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-14T00:05:51.965 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:51.980 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-14T00:05:52.041 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:52.041 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:52.041 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image13 2023-10-14T00:05:52.090 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-14T00:05:52.974 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:52.987 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-14T00:05:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image14 2023-10-14T00:05:53.097 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-14T00:05:53.982 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:53.993 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-14T00:05:54.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:54.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:54.051 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image15 2023-10-14T00:05:54.099 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-14T00:05:54.992 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:55.005 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-14T00:05:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image16 2023-10-14T00:05:55.116 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-14T00:05:55.997 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:56.009 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-14T00:05:56.070 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:56.070 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:56.070 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image17 2023-10-14T00:05:56.118 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-14T00:05:57.008 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:57.019 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-14T00:05:57.082 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:57.082 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:57.083 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image18 2023-10-14T00:05:57.130 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-14T00:05:58.025 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:58.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-14T00:05:58.095 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:58.096 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:58.096 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image19 2023-10-14T00:05:58.141 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-14T00:05:59.028 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:05:59.042 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-14T00:05:59.102 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:05:59.102 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:05:59.103 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd create -s 1G --image-feature exclusive-lock rbd/image20 2023-10-14T00:05:59.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-14T00:06:00.037 INFO:tasks.workunit.client.0.smithi105.stdout:Mirroring enabled 2023-10-14T00:06:00.053 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-14T00:06:00.117 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.117 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.118 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i = 1 )) 2023-10-14T00:06:00.118 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.118 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image1 2023-10-14T00:06:00.238 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-14T00:06:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image2 2023-10-14T00:06:00.308 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-14T00:06:00.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.309 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image3 2023-10-14T00:06:00.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-14T00:06:00.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.383 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image4 2023-10-14T00:06:00.476 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-14T00:06:00.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.477 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image5 2023-10-14T00:06:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-14T00:06:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.552 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.552 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image6 2023-10-14T00:06:00.654 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-14T00:06:00.654 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.654 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.654 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.655 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image7 2023-10-14T00:06:00.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-14T00:06:00.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image8 2023-10-14T00:06:00.877 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-14T00:06:00.877 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.877 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.878 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:00.878 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image9 2023-10-14T00:06:00.992 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-14T00:06:00.992 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:00.993 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:00.993 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image10 2023-10-14T00:06:00.998 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.111 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-14T00:06:01.112 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:01.112 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:01.112 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.113 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image11 2023-10-14T00:06:01.237 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-14T00:06:01.238 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:01.238 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:01.238 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.247 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image12 2023-10-14T00:06:01.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-14T00:06:01.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:01.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:01.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.389 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image13 2023-10-14T00:06:01.535 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-14T00:06:01.535 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:01.535 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:01.536 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.536 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image14 2023-10-14T00:06:01.702 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-14T00:06:01.703 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:01.703 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:01.703 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.707 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image15 2023-10-14T00:06:01.846 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-14T00:06:01.846 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:01.847 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:01.847 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:01.847 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image16 2023-10-14T00:06:02.010 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-14T00:06:02.010 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:02.011 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:02.011 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:02.011 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image17 2023-10-14T00:06:02.148 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-14T00:06:02.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:02.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:02.150 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image18 2023-10-14T00:06:02.150 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:02.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-14T00:06:02.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:02.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:02.282 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:02.282 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image19 2023-10-14T00:06:02.410 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-14T00:06:02.411 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:02.411 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:02.411 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:02.412 INFO:tasks.workunit.client.0.smithi105.stderr:++ sudo rbd device map rbd/image20 2023-10-14T00:06:02.518 INFO:tasks.workunit.client.0.smithi105.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-14T00:06:02.518 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T00:06:02.518 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 20 )) 2023-10-14T00:06:02.519 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:06:02.520 INFO:tasks.workunit.client.0.smithi105.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-14T00:06:02.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697241962 2023-10-14T00:06:02.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ END_TIME=1697245562 2023-10-14T00:06:02.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR= 2023-10-14T00:06:02.522 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR= 2023-10-14T00:06:02.522 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:02.522 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n '' ]] 2023-10-14T00:06:02.523 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:06:12.521 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:06:12.522 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:12.532 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:12.536 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:12.633 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:06:12.633 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:06:12.634 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:06:12.634 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:06:12.634 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:06:12.634 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:06:12.634 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:06:12.635 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:06:12.635 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:06:12.635 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:06:12.635 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:06:12.636 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:12.622+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:06:13.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3591507769 2023-10-14T00:06:13.489 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:06:13.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697241973 2023-10-14T00:06:13.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:13.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3591507769 ]] 2023-10-14T00:06:13.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3591507769 != '' ]] 2023-10-14T00:06:13.497 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3591507769 2023-10-14T00:06:15.847 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3591507769 until 2023-10-14T01:06:15.146889+0000 (3600 sec) 2023-10-14T00:06:15.905 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:06:15.907 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3591507769 2023-10-14T00:06:16.738 INFO:tasks.workunit.client.0.smithi105.stderr:listed 1 entries 2023-10-14T00:06:16.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3591507769 2023-10-14T00:06:16.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:06:26.766 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:26.767 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:26.770 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:26.784 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:06:27.631 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3591507769 2023-10-14T00:06:27.631 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:06:27.632 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697241987 2023-10-14T00:06:27.632 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:27.632 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3591507769 ]] 2023-10-14T00:06:27.633 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3591507769 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\9\1\5\0\7\7\6\9 ]] 2023-10-14T00:06:27.633 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:06:37.634 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:06:37.634 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:37.635 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:37.639 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:38.486 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3591507769 2023-10-14T00:06:38.487 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:06:38.487 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697241998 2023-10-14T00:06:38.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:38.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3591507769 ]] 2023-10-14T00:06:38.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3591507769 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\9\1\5\0\7\7\6\9 ]] 2023-10-14T00:06:38.489 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:06:48.489 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:06:48.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:06:48.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:06:48.495 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:06:49.375 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:06:49.376 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:06:49.376 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242009 2023-10-14T00:06:49.377 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:06:49.377 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:06:49.377 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\9\1\5\0\7\7\6\9 ]] 2023-10-14T00:06:49.377 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/55823494 2023-10-14T00:06:50.904 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/55823494 until 2023-10-14T01:06:50.132940+0000 (3600 sec) 2023-10-14T00:06:50.948 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:06:50.948 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/55823494 2023-10-14T00:06:51.741 INFO:tasks.workunit.client.0.smithi105.stderr:listed 2 entries 2023-10-14T00:06:51.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:06:51.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:06:59.307 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:06:59.307 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:06:59.307 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:06:59.307 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:06:59.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:06:59.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:06:59.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:06:59.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:06:59.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:06:59.309 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:06:59.309 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:06:59.309 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:06:59.304+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:07:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.016+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.016+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.016+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.016+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb17f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.016+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb16780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.016+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.020+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.020+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.033 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.032+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.033 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.032+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c30f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.038 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.036+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.036+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.040+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.043 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.040+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.044+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.044+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.048+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.051 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.048+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.054 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.052+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.052+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.056+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.056+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.062 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.060+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.062 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.060+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c30fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.077 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.076+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.078 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.076+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cd1d380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.080+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.080+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.084 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.084+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.085 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.084+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.089 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.088+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.088+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.094 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.092+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.094 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.092+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.098 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.096+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.099 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.096+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.102 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.100+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.103 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.100+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.106 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.104+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:00.106 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:07:00.104+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:07:01.773 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:01.773 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:01.777 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:01.781 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:07:02.703 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:07:02.703 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:07:02.704 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242022 2023-10-14T00:07:02.704 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:02.705 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:07:02.705 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:07:02.705 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:07:12.716 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:12.717 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:12.717 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:12.717 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:07:13.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:07:13.572 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:07:13.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242033 2023-10-14T00:07:13.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:13.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:07:13.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:07:13.574 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:07:23.575 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:23.576 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:07:23.577 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:23.582 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:24.532 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:07:24.533 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:07:24.533 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242044 2023-10-14T00:07:24.534 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:24.534 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:07:24.534 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:07:24.534 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:07:34.535 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:07:34.536 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:34.537 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:34.537 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:35.415 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:07:35.415 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:07:35.416 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242055 2023-10-14T00:07:35.416 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:35.416 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:07:35.416 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:07:35.417 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:07:45.417 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:07:45.417 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:45.417 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:45.419 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:46.385 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:07:46.386 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:07:46.387 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242066 2023-10-14T00:07:46.387 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:46.387 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:07:46.388 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:07:46.388 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:07:48.709 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:07:48.706+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:07:48.709 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:07:48.706+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:07:56.389 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:07:56.389 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:07:56.390 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:07:56.390 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:07:57.392 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/55823494 2023-10-14T00:07:57.393 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:07:57.393 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242077 2023-10-14T00:07:57.393 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:07:57.394 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/55823494 ]] 2023-10-14T00:07:57.394 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/55823494 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:07:57.394 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:07:58.973 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:07:58.966+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:07:58.973 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:07:58.966+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:07:58.973 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:07:58.966+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:08:07.395 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:08:07.395 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:07.396 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:07.401 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:08.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:08:08.328 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:08:08.328 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242088 2023-10-14T00:08:08.328 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:08.329 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1362877250 ]] 2023-10-14T00:08:08.329 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1362877250 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\5\5\8\2\3\4\9\4 ]] 2023-10-14T00:08:08.329 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/1362877250 2023-10-14T00:08:09.929 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/1362877250 until 2023-10-14T01:08:09.191771+0000 (3600 sec) 2023-10-14T00:08:09.975 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:08:09.981 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/1362877250 2023-10-14T00:08:10.752 INFO:tasks.workunit.client.0.smithi105.stderr:listed 3 entries 2023-10-14T00:08:10.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:08:10.785 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:08:20.787 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:08:20.788 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:20.792 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:20.806 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:21.707 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:08:21.708 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:08:21.709 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242101 2023-10-14T00:08:21.709 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:21.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1362877250 ]] 2023-10-14T00:08:21.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1362877250 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\3\6\2\8\7\7\2\5\0 ]] 2023-10-14T00:08:21.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:08:31.711 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:08:31.711 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:31.715 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:31.715 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:32.625 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:08:32.625 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:08:32.626 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242112 2023-10-14T00:08:32.626 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:32.627 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1362877250 ]] 2023-10-14T00:08:32.627 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1362877250 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\3\6\2\8\7\7\2\5\0 ]] 2023-10-14T00:08:32.627 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:08:42.629 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:08:42.629 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:42.645 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:42.653 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:43.511 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:08:43.512 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:08:43.512 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242123 2023-10-14T00:08:43.512 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:43.513 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1362877250 ]] 2023-10-14T00:08:43.513 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1362877250 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\3\6\2\8\7\7\2\5\0 ]] 2023-10-14T00:08:43.513 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:08:53.517 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:08:53.517 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:08:53.518 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:08:53.523 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:08:54.644 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:08:54.644 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:08:54.645 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242134 2023-10-14T00:08:54.645 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:08:54.645 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1362877250 ]] 2023-10-14T00:08:54.645 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1362877250 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\3\6\2\8\7\7\2\5\0 ]] 2023-10-14T00:08:54.646 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:09:04.648 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:09:04.658 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:04.676 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:05.606 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1362877250 2023-10-14T00:09:05.607 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:09:05.607 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242145 2023-10-14T00:09:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1362877250 ]] 2023-10-14T00:09:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1362877250 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\3\6\2\8\7\7\2\5\0 ]] 2023-10-14T00:09:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:09:15.613 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:09:15.616 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:15.616 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:15.623 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:16.487 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:09:16.487 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:09:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242156 2023-10-14T00:09:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3562409034 ]] 2023-10-14T00:09:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3562409034 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\3\6\2\8\7\7\2\5\0 ]] 2023-10-14T00:09:16.489 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3562409034 2023-10-14T00:09:18.894 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3562409034 until 2023-10-14T01:09:18.110891+0000 (3600 sec) 2023-10-14T00:09:18.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:09:18.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3562409034 2023-10-14T00:09:19.657 INFO:tasks.workunit.client.0.smithi105.stderr:listed 4 entries 2023-10-14T00:09:19.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:09:19.699 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:09:29.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:09:29.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:29.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:29.701 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:30.690 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:09:30.691 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:09:30.691 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242170 2023-10-14T00:09:30.691 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:30.691 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3562409034 ]] 2023-10-14T00:09:30.692 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3562409034 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\6\2\4\0\9\0\3\4 ]] 2023-10-14T00:09:30.692 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:09:40.693 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:40.694 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:09:40.696 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:40.697 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:41.654 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:09:41.654 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:09:41.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242181 2023-10-14T00:09:41.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:41.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3562409034 ]] 2023-10-14T00:09:41.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3562409034 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\6\2\4\0\9\0\3\4 ]] 2023-10-14T00:09:41.656 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:09:51.663 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:09:51.663 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:09:51.663 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:09:51.669 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:09:52.132 INFO:tasks.workunit.client.0.smithi105.stderr:2023-10-14T00:09:52.129+0000 7f10b962d640 0 --2- 172.21.15.105:0/3292783842 >> v2:172.21.15.105:3300/0 conn(0x7f10b4089c80 0x7f10b4091020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T00:09:52.835 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:09:52.835 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:09:52.836 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242192 2023-10-14T00:09:52.836 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:09:52.837 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3562409034 ]] 2023-10-14T00:09:52.837 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3562409034 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\6\2\4\0\9\0\3\4 ]] 2023-10-14T00:09:52.837 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:10:00.007 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:09:59.997+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:09:59.997+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.001+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.001+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d156680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.005+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.005+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d156100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.014 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.009+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.009+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d156480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.013+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.013+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.021+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.021+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c8f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.027 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.021+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.027 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.021+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c9400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.032 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.029+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.033 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.029+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c9280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.037 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.033+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.038 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.033+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.041+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.041+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.045+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.045+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.057+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.057+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c4f4300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.071 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.065+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.071 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.065+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.089 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.085+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.089 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.085+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d156e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.095 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.089+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.096 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.089+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.097+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.097+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c8f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.110 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.105+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.111 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.105+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c9580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.116 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.113+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.116 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.113+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c9780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.121 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.117+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.122 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.117+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.129 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.125+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:00.130 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:10:00.125+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d1c9a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:10:02.841 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:10:02.845 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:02.846 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:02.846 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:03.811 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:10:03.811 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:10:03.811 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242203 2023-10-14T00:10:03.812 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:03.812 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3562409034 ]] 2023-10-14T00:10:03.812 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3562409034 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\6\2\4\0\9\0\3\4 ]] 2023-10-14T00:10:03.812 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:10:12.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.297+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:10:12.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.297+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:10:12.308 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.297+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:10:12.309 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.297+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:10:12.310 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:10:12.310 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:10:12.310 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:10:12.310 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:10:12.311 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:10:12.311 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:10:12.311 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:10:12.311 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:12.301+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:10:13.812 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:10:13.813 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:13.813 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:13.818 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:14.743 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3562409034 2023-10-14T00:10:14.744 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:10:14.745 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242214 2023-10-14T00:10:14.745 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:14.745 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3562409034 ]] 2023-10-14T00:10:14.746 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3562409034 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\6\2\4\0\9\0\3\4 ]] 2023-10-14T00:10:14.746 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:10:24.751 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:10:24.753 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:24.759 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:24.768 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:25.627 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:10:25.636 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:10:25.637 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242225 2023-10-14T00:10:25.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:25.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1238328416 ]] 2023-10-14T00:10:25.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1238328416 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\5\6\2\4\0\9\0\3\4 ]] 2023-10-14T00:10:25.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/1238328416 2023-10-14T00:10:27.957 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/1238328416 until 2023-10-14T01:10:27.256274+0000 (3600 sec) 2023-10-14T00:10:27.999 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:10:28.008 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/1238328416 2023-10-14T00:10:28.760 INFO:tasks.workunit.client.0.smithi105.stderr:listed 5 entries 2023-10-14T00:10:28.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:10:28.785 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:10:32.130 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:10:32.130 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:10:32.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:10:32.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:10:32.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:10:32.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:10:32.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:10:32.132 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:10:32.132 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:10:32.132 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:10:32.132 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:10:32.132 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:10:32.133 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:10:32.121+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:10:38.786 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:10:38.808 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:39.686 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:10:39.686 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:10:39.687 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242239 2023-10-14T00:10:39.687 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:39.687 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1238328416 ]] 2023-10-14T00:10:39.687 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1238328416 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\2\3\8\3\2\8\4\1\6 ]] 2023-10-14T00:10:39.687 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:10:40.010 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:10:40.005+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:10:40.010 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:10:40.005+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:10:49.689 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:10:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:10:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:10:49.707 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:10:50.672 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:10:50.672 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:10:50.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242250 2023-10-14T00:10:50.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:10:50.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1238328416 ]] 2023-10-14T00:10:50.674 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1238328416 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\2\3\8\3\2\8\4\1\6 ]] 2023-10-14T00:10:50.674 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:11:00.002 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.000+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.002 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.000+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.004+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.004+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.008+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.008+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.012+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.013 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.012+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.016+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.016+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.020 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.020+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.020 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.020+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.024+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.024+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.028 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.028+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.028+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.031 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.028+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.031 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.028+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.040 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.040+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.040 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.040+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.040+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.043 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.040+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d26f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.045 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.044+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.045 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.044+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d26f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.048+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.048+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.054 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.052+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.052+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.058 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.056+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.058 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.056+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.064+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.064+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.068+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.068+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.076+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.076+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.083 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.080+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.083 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.080+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04fd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.088 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.088+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.089 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:11:00.088+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:11:00.677 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:11:00.678 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:00.679 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:00.682 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:01.545 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:11:01.553 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:11:01.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242261 2023-10-14T00:11:01.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:01.555 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1238328416 ]] 2023-10-14T00:11:01.555 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1238328416 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\2\3\8\3\2\8\4\1\6 ]] 2023-10-14T00:11:01.555 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:11:11.564 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:11:11.564 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:11.564 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:11.573 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:12.509 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:11:12.509 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:11:12.510 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242272 2023-10-14T00:11:12.510 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:12.510 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1238328416 ]] 2023-10-14T00:11:12.511 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1238328416 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\2\3\8\3\2\8\4\1\6 ]] 2023-10-14T00:11:12.511 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:11:22.519 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:11:22.520 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:22.523 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:22.527 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:23.451 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1238328416 2023-10-14T00:11:23.451 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:11:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242283 2023-10-14T00:11:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1238328416 ]] 2023-10-14T00:11:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1238328416 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\2\3\8\3\2\8\4\1\6 ]] 2023-10-14T00:11:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:11:33.453 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:11:33.454 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:33.454 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:33.455 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:34.325 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:11:34.326 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:11:34.326 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242294 2023-10-14T00:11:34.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:34.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/396685582 ]] 2023-10-14T00:11:34.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/396685582 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\2\3\8\3\2\8\4\1\6 ]] 2023-10-14T00:11:34.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/396685582 2023-10-14T00:11:35.982 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/396685582 until 2023-10-14T01:11:35.175586+0000 (3600 sec) 2023-10-14T00:11:36.018 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:11:36.018 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/396685582 2023-10-14T00:11:36.788 INFO:tasks.workunit.client.0.smithi105.stderr:listed 6 entries 2023-10-14T00:11:36.821 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:11:36.821 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:11:46.823 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:11:46.823 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:46.824 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:46.835 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:47.666 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:11:47.667 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:11:47.672 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242307 2023-10-14T00:11:47.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:47.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/396685582 ]] 2023-10-14T00:11:47.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/396685582 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\6\6\8\5\5\8\2 ]] 2023-10-14T00:11:47.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:11:57.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:11:57.684 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:11:57.687 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:11:57.692 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:11:58.538 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:11:58.539 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:11:58.539 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242318 2023-10-14T00:11:58.540 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:11:58.540 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/396685582 ]] 2023-10-14T00:11:58.540 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/396685582 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\6\6\8\5\5\8\2 ]] 2023-10-14T00:11:58.540 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:12:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.000+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.000+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d26ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.004+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.004+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d26e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.008+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.008+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66ceed880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.012+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.012+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d26f580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.020+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.020+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d156e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.026 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.024+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.026 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.024+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d04fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.028+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.028+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d04ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.036 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.036+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.037 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.036+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d04f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.040+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.040+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d382a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.044+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.044+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.053 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.052+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.053 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.052+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d04f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.060 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.060+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.060+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d04f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.064+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.064+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d382d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.072+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.077 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.072+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d382f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.086 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.084+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.087 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.084+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.091 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.088+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.091 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.092+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.096 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.096+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.096 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.096+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.100 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.100+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.100 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.100+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.105 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.104+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.105 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.104+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.109 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.108+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:00.110 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:12:00.108+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:12:08.543 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:12:08.546 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:08.547 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:08.558 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:09.488 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:12:09.489 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:12:09.489 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242329 2023-10-14T00:12:09.489 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:09.490 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/396685582 ]] 2023-10-14T00:12:09.490 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/396685582 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\6\6\8\5\5\8\2 ]] 2023-10-14T00:12:09.490 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:12:19.491 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:12:19.502 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:19.508 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:20.490 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:12:20.490 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:12:20.491 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242340 2023-10-14T00:12:20.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:20.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/396685582 ]] 2023-10-14T00:12:20.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/396685582 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\6\6\8\5\5\8\2 ]] 2023-10-14T00:12:20.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:12:30.512 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:12:30.514 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:30.518 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:30.525 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:31.475 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/396685582 2023-10-14T00:12:31.477 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:12:31.485 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242351 2023-10-14T00:12:31.485 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:31.486 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/396685582 ]] 2023-10-14T00:12:31.486 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/396685582 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\6\6\8\5\5\8\2 ]] 2023-10-14T00:12:31.486 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:12:41.491 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:12:41.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:41.494 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:41.502 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:42.460 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:12:42.460 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:12:42.461 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242362 2023-10-14T00:12:42.461 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:42.461 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3267454688 ]] 2023-10-14T00:12:42.462 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3267454688 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\6\6\8\5\5\8\2 ]] 2023-10-14T00:12:42.462 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3267454688 2023-10-14T00:12:43.962 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3267454688 until 2023-10-14T01:12:43.142287+0000 (3600 sec) 2023-10-14T00:12:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:12:44.007 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3267454688 2023-10-14T00:12:44.843 INFO:tasks.workunit.client.0.smithi105.stderr:listed 7 entries 2023-10-14T00:12:44.890 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:12:44.890 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:12:54.892 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:12:54.901 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:12:54.903 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:12:54.910 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:12:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:12:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:12:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242375 2023-10-14T00:12:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:12:55.724 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3267454688 ]] 2023-10-14T00:12:55.724 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3267454688 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\2\6\7\4\5\4\6\8\8 ]] 2023-10-14T00:12:55.724 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:13:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.003+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.003+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1ca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.005 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.003+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.005 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.003+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb12180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.007+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.007+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb12080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.011+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.011+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb12a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.015+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.015+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66ceecb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.019+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.019+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d1c8680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.026 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.023+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.026 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.023+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c7f6e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.028 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.027+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.028 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.027+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c4f5c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.032 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.031+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.032 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.031+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb13f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.037 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.035+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.038 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.035+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb13e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.039+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.039+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1ca80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.043+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.043+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d04eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.051+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.051 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.051+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d1c8b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.059+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.060 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.059+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d1c9c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.074 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.071+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.074 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.071+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb12a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.079 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.075+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.079 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.075+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d383580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.083 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.083+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.083 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.083+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1cb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.092 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.087+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.092 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.087+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c4f5d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.092 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.091+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.092 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.091+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66ceeda80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.094 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.095+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:00.095 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:00.095+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c4f5780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:13:03.062 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:13:03.063+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:13:05.734 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:05.738 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:13:05.741 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:05.751 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:13:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:13:06.735 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242386 2023-10-14T00:13:06.735 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:06.735 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3267454688 ]] 2023-10-14T00:13:06.735 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3267454688 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\2\6\7\4\5\4\6\8\8 ]] 2023-10-14T00:13:06.736 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:13:16.737 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:13:16.737 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:16.738 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:16.746 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:17.822 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:13:17.823 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:13:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242397 2023-10-14T00:13:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:17.826 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3267454688 ]] 2023-10-14T00:13:17.826 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3267454688 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\2\6\7\4\5\4\6\8\8 ]] 2023-10-14T00:13:17.826 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:13:27.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:13:27.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:27.832 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:27.832 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:28.595 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:13:28.595 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:13:28.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242408 2023-10-14T00:13:28.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:28.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3267454688 ]] 2023-10-14T00:13:28.597 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3267454688 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\2\6\7\4\5\4\6\8\8 ]] 2023-10-14T00:13:28.597 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:13:38.601 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:13:38.606 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:38.614 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:39.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3267454688 2023-10-14T00:13:39.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:13:39.493 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242419 2023-10-14T00:13:39.493 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:39.493 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3267454688 ]] 2023-10-14T00:13:39.493 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3267454688 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\2\6\7\4\5\4\6\8\8 ]] 2023-10-14T00:13:39.494 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:13:49.494 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:13:49.498 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:13:49.498 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:13:49.509 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:13:50.494 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:13:50.495 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:13:50.495 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242430 2023-10-14T00:13:50.495 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:13:50.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:13:50.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\2\6\7\4\5\4\6\8\8 ]] 2023-10-14T00:13:50.496 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/390132398 2023-10-14T00:13:53.009 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/390132398 until 2023-10-14T01:13:52.190864+0000 (3600 sec) 2023-10-14T00:13:53.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:13:53.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/390132398 2023-10-14T00:13:53.801 INFO:tasks.workunit.client.0.smithi105.stderr:listed 8 entries 2023-10-14T00:13:53.826 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:13:53.827 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:14:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:59.998+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:13:59.998+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb12880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.006+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.006+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1d400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.006+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.006+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1ce00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.010+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.010+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.018+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.018+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c4f4680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.031 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.030+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.032 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.030+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d04fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.041 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.038+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.042 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.038+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d383d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.046+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.047 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.046+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d04f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.046+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.046+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d1c9100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.058+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.062 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.058+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d383b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.067 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.066+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.067 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.066+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d383f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.078 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.074+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.078 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.074+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d383f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.086+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.086+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.100 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.098+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.100 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.098+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.105 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.102+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.105 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.102+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.116 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.114+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.116 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.114+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.128 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.126+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.128 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.126+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.134 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.130+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.134 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.130+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.142 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.138+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.142 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.138+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a1780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.149 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.146+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:00.149 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:14:00.146+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a1600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:14:03.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:14:03.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:03.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:03.841 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:04.748 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:14:04.749 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:14:04.750 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242444 2023-10-14T00:14:04.750 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:04.750 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:14:04.751 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:14:04.751 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:14:14.758 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:14.758 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:14.769 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:14.774 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:14:15.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:14:15.617 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:14:15.618 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242455 2023-10-14T00:14:15.618 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:15.618 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:14:15.619 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:14:15.619 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:14:25.620 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:14:25.620 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:25.623 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:25.627 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:26.566 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:14:26.566 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:14:26.567 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242466 2023-10-14T00:14:26.567 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:26.567 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:14:26.567 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:14:26.568 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:14:26.584 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:14:26.582+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:14:26.584 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:14:26.582+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:14:26.584 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:14:26.582+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:14:36.568 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:14:36.569 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:36.576 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:36.576 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:14:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:14:37.634 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242477 2023-10-14T00:14:37.635 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:37.635 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:14:37.635 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:14:37.635 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:14:41.924 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:14:41.922+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:14:41.924 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:14:41.922+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:14:47.642 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:47.643 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:47.650 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:14:47.670 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:48.708 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:14:48.709 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:14:48.709 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242488 2023-10-14T00:14:48.709 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:48.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:14:48.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:14:48.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:14:58.714 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:14:58.714 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:14:58.716 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:14:58.720 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:14:59.740 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/390132398 2023-10-14T00:14:59.740 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:14:59.740 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242499 2023-10-14T00:14:59.741 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:14:59.741 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/390132398 ]] 2023-10-14T00:14:59.741 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/390132398 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:14:59.741 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:15:01.293 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:01.285+0000 7f5baaa31640 -1 reset not still connected to 0x562d42a6d110 2023-10-14T00:15:08.271 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:15:08.272 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:15:08.272 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:15:08.272 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:15:08.272 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:15:08.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:15:08.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:15:08.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:15:08.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:15:08.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:15:08.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:15:08.274 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:15:08.274 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:08.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:15:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:09.745 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:15:10.651 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:15:10.651 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:15:10.652 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242510 2023-10-14T00:15:10.652 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:10.652 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/2655824907 ]] 2023-10-14T00:15:10.653 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/2655824907 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\9\0\1\3\2\3\9\8 ]] 2023-10-14T00:15:10.653 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/2655824907 2023-10-14T00:15:13.000 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/2655824907 until 2023-10-14T01:15:12.203990+0000 (3600 sec) 2023-10-14T00:15:13.037 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:15:13.042 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/2655824907 2023-10-14T00:15:13.848 INFO:tasks.workunit.client.0.smithi105.stderr:listed 9 entries 2023-10-14T00:15:13.884 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:15:13.884 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:15:23.890 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:15:23.892 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:23.894 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:23.904 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:24.879 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:15:24.879 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:15:24.879 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242524 2023-10-14T00:15:24.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:24.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/2655824907 ]] 2023-10-14T00:15:24.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/2655824907 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\2\6\5\5\8\2\4\9\0\7 ]] 2023-10-14T00:15:24.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:15:34.881 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:34.882 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:34.882 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:34.890 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:15:35.813 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:15:35.814 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:15:35.814 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242535 2023-10-14T00:15:35.814 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:35.815 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/2655824907 ]] 2023-10-14T00:15:35.815 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/2655824907 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\2\6\5\5\8\2\4\9\0\7 ]] 2023-10-14T00:15:35.815 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:15:45.823 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:45.824 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:15:45.824 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:45.832 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:46.913 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:15:46.913 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:15:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242546 2023-10-14T00:15:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/2655824907 ]] 2023-10-14T00:15:46.915 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/2655824907 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\2\6\5\5\8\2\4\9\0\7 ]] 2023-10-14T00:15:46.915 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:15:56.916 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:15:56.919 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:15:56.927 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:15:56.933 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:15:57.887 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:15:57.887 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:15:57.887 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242557 2023-10-14T00:15:57.888 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:15:57.888 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/2655824907 ]] 2023-10-14T00:15:57.888 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/2655824907 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\2\6\5\5\8\2\4\9\0\7 ]] 2023-10-14T00:15:57.888 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:15:59.251 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:15:59.245+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:16:07.889 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:16:07.890 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:07.890 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:07.890 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:08.778 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/2655824907 2023-10-14T00:16:08.778 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:16:08.778 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242568 2023-10-14T00:16:08.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:08.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/2655824907 ]] 2023-10-14T00:16:08.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/2655824907 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\2\6\5\5\8\2\4\9\0\7 ]] 2023-10-14T00:16:08.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:16:18.781 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:18.781 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:18.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:18.809 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:16:19.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:16:19.781 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:16:19.786 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242579 2023-10-14T00:16:19.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:19.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3438481267 ]] 2023-10-14T00:16:19.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3438481267 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\2\6\5\5\8\2\4\9\0\7 ]] 2023-10-14T00:16:19.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3438481267 2023-10-14T00:16:22.041 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3438481267 until 2023-10-14T01:16:21.295630+0000 (3600 sec) 2023-10-14T00:16:22.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:16:22.080 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3438481267 2023-10-14T00:16:22.801 INFO:tasks.workunit.client.0.smithi105.stderr:listed 10 entries 2023-10-14T00:16:22.856 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:16:22.857 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:16:25.738 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:16:25.740+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:16:25.738 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:16:25.740+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:16:25.738 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:16:25.740+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:16:32.863 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:16:32.863 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:32.865 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:32.887 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:33.673 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:16:33.674 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:16:33.678 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242593 2023-10-14T00:16:33.678 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:33.678 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3438481267 ]] 2023-10-14T00:16:33.678 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3438481267 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\4\3\8\4\8\1\2\6\7 ]] 2023-10-14T00:16:33.678 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:16:43.682 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:43.684 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:16:43.685 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:43.698 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:44.642 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:16:44.643 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:16:44.643 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242604 2023-10-14T00:16:44.643 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:44.643 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3438481267 ]] 2023-10-14T00:16:44.644 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3438481267 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\4\3\8\4\8\1\2\6\7 ]] 2023-10-14T00:16:44.644 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:16:54.645 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:16:54.646 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:16:54.646 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:16:54.661 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:16:55.564 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:16:55.564 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:16:55.566 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242615 2023-10-14T00:16:55.566 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:16:55.566 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3438481267 ]] 2023-10-14T00:16:55.566 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3438481267 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\4\3\8\4\8\1\2\6\7 ]] 2023-10-14T00:16:55.566 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:17:00.000 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.000+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.000+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a1680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.003 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.004+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.003 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.004+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a0000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.008+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.008+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a1480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.008+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.008+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a1500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.012+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.012+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a0d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.013 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.016+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.013 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.016+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c4f4280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.020+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.022 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.020+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c4f4480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.024+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.025 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.024+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.036+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.036+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.049 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.048+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.048+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.060 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.060+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.060 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.060+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.066 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.064+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.066 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.068+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.075 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.076+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.076+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.079 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.080+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.079 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.080+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.085 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.084+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.085 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.084+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d774f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.093 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.092+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.095 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.092+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d775180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.100 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.100+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.100+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d775380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.106 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.108+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.107 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.108+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d775580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.112 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.112+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.112 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.112+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d775780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.118 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.120+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:00.118 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:17:00.120+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d775980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:17:05.567 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:17:05.572 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:05.577 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:05.581 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242626 2023-10-14T00:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3438481267 ]] 2023-10-14T00:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3438481267 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\4\3\8\4\8\1\2\6\7 ]] 2023-10-14T00:17:06.528 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:17:16.537 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:17:16.541 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:16.543 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:16.549 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:17.655 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:17:17.652+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:17:17.657 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3438481267 2023-10-14T00:17:17.657 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:17:17.659 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242637 2023-10-14T00:17:17.659 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:17.659 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3438481267 ]] 2023-10-14T00:17:17.659 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3438481267 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\4\3\8\4\8\1\2\6\7 ]] 2023-10-14T00:17:17.660 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:17:27.663 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:17:27.663 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:27.663 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:27.679 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:28.602 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:17:28.603 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:17:28.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242648 2023-10-14T00:17:28.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:28.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3800667290 ]] 2023-10-14T00:17:28.605 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3800667290 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\4\3\8\4\8\1\2\6\7 ]] 2023-10-14T00:17:28.605 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3800667290 2023-10-14T00:17:31.030 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3800667290 until 2023-10-14T01:17:30.331648+0000 (3600 sec) 2023-10-14T00:17:31.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:17:31.061 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3800667290 2023-10-14T00:17:31.876 INFO:tasks.workunit.client.0.smithi105.stderr:listed 11 entries 2023-10-14T00:17:31.904 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:17:31.905 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:17:35.065 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:17:35.063+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:17:35.066 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:17:35.063+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:17:35.066 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:17:35.063+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:17:41.910 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:17:41.910 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:41.915 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:41.920 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:42.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:17:42.755 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:17:42.755 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242662 2023-10-14T00:17:42.755 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:42.755 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3800667290 ]] 2023-10-14T00:17:42.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3800667290 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\8\0\0\6\6\7\2\9\0 ]] 2023-10-14T00:17:42.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:17:52.757 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:17:52.758 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:17:52.758 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:17:52.762 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:17:53.720 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:17:53.721 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:17:53.721 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242673 2023-10-14T00:17:53.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:17:53.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3800667290 ]] 2023-10-14T00:17:53.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3800667290 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\8\0\0\6\6\7\2\9\0 ]] 2023-10-14T00:17:53.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:18:00.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.007+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.006 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.007+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.007 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.007+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d04eb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.007 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.007+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d382880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.011+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.011+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a1580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.014 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.015+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.014 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.015+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d04f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.019+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.019+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb17380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.023 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.023+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.023 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.023+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.027 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.027+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.028 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.027+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.031+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.031+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.033 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.035+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.033 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.035+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71ac00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.044 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.043+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.044 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.043+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71aa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.044 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.043+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.044 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.043+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.054 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.055+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.055+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.059+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.059+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.063 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.063+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.063 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.063+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.068 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.067+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.068 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.067+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.072 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.071+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.072 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.071+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.077 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.075+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.077 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.075+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.083+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.083+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.086 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.087+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.086 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.087+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71af00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.091+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:00.091+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:18:03.737 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:18:03.737 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:03.741 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:03.748 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:04.581 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:18:04.582 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:18:04.583 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242684 2023-10-14T00:18:04.583 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:04.583 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3800667290 ]] 2023-10-14T00:18:04.584 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3800667290 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\8\0\0\6\6\7\2\9\0 ]] 2023-10-14T00:18:04.584 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:18:04.659 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:04.659+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:18:04.659 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:04.659+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:18:04.660 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:04.659+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:18:04.661 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:04.663+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:18:14.583 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:18:14.584 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:14.585 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:14.606 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:15.518 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:18:15.519 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:18:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242695 2023-10-14T00:18:15.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:15.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3800667290 ]] 2023-10-14T00:18:15.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3800667290 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\8\0\0\6\6\7\2\9\0 ]] 2023-10-14T00:18:15.521 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:18:22.575 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:22.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:18:22.575 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:22.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:18:22.576 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:18:22.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:18:25.522 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:18:25.523 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:25.523 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:25.529 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:26.293 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3800667290 2023-10-14T00:18:26.294 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:18:26.295 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242706 2023-10-14T00:18:26.295 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:26.295 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3800667290 ]] 2023-10-14T00:18:26.295 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3800667290 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\8\0\0\6\6\7\2\9\0 ]] 2023-10-14T00:18:26.295 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:18:36.300 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:36.301 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:18:36.301 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:36.319 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:37.238 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:18:37.239 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:18:37.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242717 2023-10-14T00:18:37.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:37.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3372654654 ]] 2023-10-14T00:18:37.241 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3372654654 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\8\0\0\6\6\7\2\9\0 ]] 2023-10-14T00:18:37.241 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3372654654 2023-10-14T00:18:39.061 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3372654654 until 2023-10-14T01:18:38.280526+0000 (3600 sec) 2023-10-14T00:18:39.112 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:18:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3372654654 2023-10-14T00:18:40.083 INFO:tasks.workunit.client.0.smithi105.stderr:listed 12 entries 2023-10-14T00:18:40.123 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:18:40.124 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:18:50.125 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:18:50.126 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:18:50.127 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:18:50.145 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:18:50.979 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:18:50.980 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:18:50.980 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242730 2023-10-14T00:18:50.980 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:18:50.981 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3372654654 ]] 2023-10-14T00:18:50.981 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3372654654 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\3\7\2\6\5\4\6\5\4 ]] 2023-10-14T00:18:50.981 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:19:00.000 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:59.998+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:18:59.998+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb13e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.003 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.002+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.003 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.002+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c7f6800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.010+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.010+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d056c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.014+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.014+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c7f7000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.023 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.022+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.022+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d057f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.030+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.030+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d383780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.034 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.034+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.034+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a0a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.038+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.038+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb16b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.043 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.042+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.044 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.042+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a0c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.048 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.046+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.048 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.046+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.050+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.050+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.057 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.054+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.057 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.054+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d382780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.062+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.062+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d775c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.062+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.062+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d1c8680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.068 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.066+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.068 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.066+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d04f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.078+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.078+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.086 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.086+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.086 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.086+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71ba80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.092 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.090+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.093 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.090+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71aa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.099 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.098+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.099 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.098+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.103 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.102+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.104 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:00.102+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71b300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:19:00.992 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:00.992 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:19:01.002 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:01.005 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:01.978 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:19:01.979 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:19:01.982 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242741 2023-10-14T00:19:01.982 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:01.982 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3372654654 ]] 2023-10-14T00:19:01.982 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3372654654 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\3\7\2\6\5\4\6\5\4 ]] 2023-10-14T00:19:01.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:19:11.984 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:19:11.985 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:11.985 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:11.985 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:12.871 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:19:12.871 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:19:12.872 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242752 2023-10-14T00:19:12.872 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:12.872 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3372654654 ]] 2023-10-14T00:19:12.872 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3372654654 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\3\7\2\6\5\4\6\5\4 ]] 2023-10-14T00:19:12.872 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:19:15.245 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:19:15.242+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:19:15.245 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:19:15.242+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:19:15.245 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:19:15.242+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:19:22.874 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:22.874 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:22.878 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:22.880 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:19:23.775 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:19:23.775 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:19:23.776 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242763 2023-10-14T00:19:23.776 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:23.776 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3372654654 ]] 2023-10-14T00:19:23.776 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3372654654 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\3\7\2\6\5\4\6\5\4 ]] 2023-10-14T00:19:23.777 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:19:33.778 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:19:33.779 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:33.780 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:33.801 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:34.777 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3372654654 2023-10-14T00:19:34.778 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:19:34.778 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242774 2023-10-14T00:19:34.778 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:34.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3372654654 ]] 2023-10-14T00:19:34.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3372654654 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\3\7\2\6\5\4\6\5\4 ]] 2023-10-14T00:19:34.779 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:19:44.780 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:44.780 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:19:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:45.817 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:19:45.817 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:19:45.819 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242785 2023-10-14T00:19:45.819 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:45.819 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/115563815 ]] 2023-10-14T00:19:45.820 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/115563815 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\3\7\2\6\5\4\6\5\4 ]] 2023-10-14T00:19:45.820 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/115563815 2023-10-14T00:19:46.665 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:19:46.661+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:19:46.666 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:19:46.665+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:19:46.667 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:19:46.665+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:19:46.670 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:19:46.665+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:19:48.085 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/115563815 until 2023-10-14T01:19:47.292604+0000 (3600 sec) 2023-10-14T00:19:48.133 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:19:48.147 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/115563815 2023-10-14T00:19:48.940 INFO:tasks.workunit.client.0.smithi105.stderr:listed 13 entries 2023-10-14T00:19:48.975 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:19:48.976 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:19:58.976 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:19:58.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:19:58.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:19:58.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:19:59.946 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:19:59.946 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:19:59.947 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242799 2023-10-14T00:19:59.947 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:19:59.948 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/115563815 ]] 2023-10-14T00:19:59.948 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/115563815 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\1\5\5\6\3\8\1\5 ]] 2023-10-14T00:19:59.948 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:20:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:59.997+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:19:59.997+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d382500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.001+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.001+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d057680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.005+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.005+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c4f5900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.009+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.009+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d057a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.020 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.017+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.017+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66c2d9a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.021+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.021+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d26ff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.028 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.025+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.025+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cb12e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.037+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.037+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.048 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.045+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.048 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.045+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.049+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.049+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d774480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.061+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.061+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d775100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.069 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.069+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.069+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66cb13e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.073+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.073+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71bc00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.080 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.077+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.077+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.091 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.089+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.092 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.089+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71b080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.096 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.093+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.097 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.093+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.105 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.105+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.106 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.105+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d7f2000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.108 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.105+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.108 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.105+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d7f2200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.111 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.109+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.111 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.109+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d7f2400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.113 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.113+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:00.113 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:20:00.113+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d7f2600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:20:09.394 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:20:09.393+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:20:09.956 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:20:09.956 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:09.956 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:09.957 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:10.811 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:20:10.812 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:20:10.813 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242810 2023-10-14T00:20:10.813 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:10.813 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/115563815 ]] 2023-10-14T00:20:10.814 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/115563815 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\1\5\5\6\3\8\1\5 ]] 2023-10-14T00:20:10.814 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:20:20.821 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:20:20.821 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:20.821 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:20.829 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:21.767 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:20:21.768 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:20:21.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242821 2023-10-14T00:20:21.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:21.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/115563815 ]] 2023-10-14T00:20:21.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/115563815 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\1\5\5\6\3\8\1\5 ]] 2023-10-14T00:20:21.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:20:23.808 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:20:23.808 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:20:23.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:20:23.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:20:23.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:20:23.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:20:23.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:23.805+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:20:26.567 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:20:26.565+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:20:26.567 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:20:26.565+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:20:26.567 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:20:26.565+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:20:31.770 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:31.772 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:31.793 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:20:31.798 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:32.628 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:20:32.628 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:20:32.629 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242832 2023-10-14T00:20:32.629 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:32.630 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/115563815 ]] 2023-10-14T00:20:32.630 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/115563815 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\1\5\5\6\3\8\1\5 ]] 2023-10-14T00:20:32.630 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:20:36.101 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:20:36.101 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:20:36.102 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:20:36.102 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:20:36.102 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:20:36.102 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:20:36.102 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:20:36.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:20:36.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:20:36.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:20:36.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:20:36.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:20:36.104 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:36.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:20:42.631 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:42.634 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:42.640 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:20:42.652 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:43.553 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/115563815 2023-10-14T00:20:43.553 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:20:43.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242843 2023-10-14T00:20:43.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:43.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/115563815 ]] 2023-10-14T00:20:43.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/115563815 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\1\5\5\6\3\8\1\5 ]] 2023-10-14T00:20:43.554 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:20:53.563 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:20:53.564 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:20:53.567 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:20:53.573 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:20:54.581 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:20:54.581 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:20:54.582 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242854 2023-10-14T00:20:54.582 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:20:54.582 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/995254717 ]] 2023-10-14T00:20:54.582 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/995254717 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\1\5\5\6\3\8\1\5 ]] 2023-10-14T00:20:54.583 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/995254717 2023-10-14T00:20:57.089 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/995254717 until 2023-10-14T01:20:56.302091+0000 (3600 sec) 2023-10-14T00:20:57.125 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:20:57.138 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/995254717 2023-10-14T00:20:57.859 INFO:tasks.workunit.client.0.smithi105.stderr:listed 14 entries 2023-10-14T00:20:57.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:20:57.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:20:59.878 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.876+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:20:59.878 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.876+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:20:59.878 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.876+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:20:59.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.884+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:20:59.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.884+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:20:59.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.884+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:20:59.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.884+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:20:59.885 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.884+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:20:59.885 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:20:59.884+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:21:00.010 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.011 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d4a0b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d04f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d26f780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66ceec300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.012 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.013 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.012+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1cd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.016+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.016+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d056f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.022 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.024+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.023 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.024+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d775980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.028+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.028+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d775800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.034 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.036+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.034 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.036+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.038 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.040+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.040+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f2700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.044 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.044+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.045 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.044+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f2980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.052+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.052+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d775d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.056+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.056+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f2d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.060+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.060+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f2f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.064+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.064 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.064+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.072+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.072+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.080 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.080+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.080+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.087 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.088+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.088 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.088+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.096 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.096+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.097 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.096+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.104+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:21:00.104+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:21:07.887 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:21:07.887 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:07.888 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:07.902 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:10.152 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:21:10.156 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:21:10.219 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242870 2023-10-14T00:21:10.219 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:10.219 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/995254717 ]] 2023-10-14T00:21:10.219 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/995254717 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\9\9\5\2\5\4\7\1\7 ]] 2023-10-14T00:21:10.220 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:21:20.610 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:20.611 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:20.611 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:20.639 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:21:21.621 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:21:21.621 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:21:21.623 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242881 2023-10-14T00:21:21.623 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:21.624 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/995254717 ]] 2023-10-14T00:21:21.625 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/995254717 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\9\9\5\2\5\4\7\1\7 ]] 2023-10-14T00:21:21.625 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:21:26.972 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:21:26.968+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:21:26.973 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:21:26.972+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:21:26.974 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:21:26.972+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:21:26.988 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:21:26.980+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:21:31.637 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:21:31.639 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:31.640 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:31.640 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:32.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:21:32.767 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:21:32.768 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242892 2023-10-14T00:21:32.768 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:32.768 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/995254717 ]] 2023-10-14T00:21:32.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/995254717 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\9\9\5\2\5\4\7\1\7 ]] 2023-10-14T00:21:32.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:21:42.833 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:21:42.834 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:42.836 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:42.842 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:43.844 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:21:43.845 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:21:43.845 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242903 2023-10-14T00:21:43.845 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:43.846 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/995254717 ]] 2023-10-14T00:21:43.846 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/995254717 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\9\9\5\2\5\4\7\1\7 ]] 2023-10-14T00:21:43.846 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:21:53.848 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:21:53.849 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:21:53.852 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:21:53.856 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:21:54.940 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/995254717 2023-10-14T00:21:54.944 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:21:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242914 2023-10-14T00:21:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:21:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/995254717 ]] 2023-10-14T00:21:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/995254717 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\9\9\5\2\5\4\7\1\7 ]] 2023-10-14T00:21:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:22:04.952 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:22:04.957 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:04.962 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:04.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:05.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:22:05.861 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:22:05.862 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242925 2023-10-14T00:22:05.862 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:05.862 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:22:05.862 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\9\9\5\2\5\4\7\1\7 ]] 2023-10-14T00:22:05.863 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/4114602057 2023-10-14T00:22:07.715 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:22:07.711+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:22:08.128 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/4114602057 until 2023-10-14T01:22:07.393006+0000 (3600 sec) 2023-10-14T00:22:08.175 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/4114602057 2023-10-14T00:22:08.176 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:22:08.939 INFO:tasks.workunit.client.0.smithi105.stderr:listed 15 entries 2023-10-14T00:22:08.962 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:22:08.963 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:22:18.969 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:22:18.969 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:18.973 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:18.986 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:20.100 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:22:20.100 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:22:20.101 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242940 2023-10-14T00:22:20.101 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:20.101 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:22:20.101 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:22:20.102 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:22:30.105 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:22:30.107 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:30.107 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:30.110 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:31.047 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:22:31.048 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:22:31.048 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242951 2023-10-14T00:22:31.048 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:31.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:22:31.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:22:31.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:22:41.055 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:22:41.060 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:41.068 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:41.077 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:22:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:22:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242961 2023-10-14T00:22:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:22:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:22:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:22:51.993 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:22:52.001 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:22:52.006 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:22:52.022 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:22:52.935 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:22:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:22:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242972 2023-10-14T00:22:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:22:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:22:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:22:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:23:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.003+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.003+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66cb13600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.003 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.003+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.003+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.011+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.008 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.011+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71b200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.019+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.017 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.019+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71aa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.023 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.023+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.024 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.023+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d71a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.032 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.035+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.032 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.035+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d774800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.035+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.035+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.048 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.051+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.048 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.051+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d774c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.051+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.051+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.055+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.055+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.059+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.059 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.059+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.066 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.067+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.066 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.067+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d774280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.069 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.071+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.069 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.071+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d850000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.078 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.079+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.078 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.079+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d850200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.083+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.083+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d850400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.085 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.087+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.085 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.087+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d850600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.091+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.091+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d850800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.094 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.095+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.094 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.095+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d850a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.098 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.099+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.098 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.099+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d850c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.103+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:00.101 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:23:00.103+0000 7f93c57a7640 -1 librbd::ImageState: 0x55a66d850e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:23:02.947 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:02.963 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:02.964 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:02.967 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:23:03.974 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:23:03.974 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:23:03.975 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242983 2023-10-14T00:23:03.975 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:03.975 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:23:03.976 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:23:03.976 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:23:13.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:13.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:13.979 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:23:13.980 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:15.083 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/4114602057 2023-10-14T00:23:15.083 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:23:15.084 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697242995 2023-10-14T00:23:15.084 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:15.085 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/4114602057 ]] 2023-10-14T00:23:15.085 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/4114602057 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:23:15.085 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:23:15.901 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:23:15.903+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:23:15.902 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:23:15.903+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:23:20.384 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:23:20.383+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:23:25.086 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:25.086 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:25.087 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:23:25.121 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:26.053 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:23:26.054 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:23:26.054+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:23:26.055 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:23:26.058+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:23:26.058 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:23:26.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243006 2023-10-14T00:23:26.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:26.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/654405172 ]] 2023-10-14T00:23:26.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/654405172 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\4\1\1\4\6\0\2\0\5\7 ]] 2023-10-14T00:23:26.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/654405172 2023-10-14T00:23:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/654405172 until 2023-10-14T01:23:27.405586+0000 (3600 sec) 2023-10-14T00:23:28.188 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/654405172 2023-10-14T00:23:28.190 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:23:28.945 INFO:tasks.workunit.client.0.smithi105.stderr:listed 16 entries 2023-10-14T00:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:23:30.405 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:23:30.406+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:23:30.405 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:23:30.406+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:23:30.405 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:23:30.406+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:23:38.998 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:38.998 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:23:38.999 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:39.008 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:39.943 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:23:39.944 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:23:39.944 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243019 2023-10-14T00:23:39.945 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:39.945 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/654405172 ]] 2023-10-14T00:23:39.945 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/654405172 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\6\5\4\4\0\5\1\7\2 ]] 2023-10-14T00:23:39.945 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:23:49.947 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:23:49.948 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:23:49.949 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:23:49.961 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:23:51.136 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:23:51.137 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:23:51.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243031 2023-10-14T00:23:51.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:23:51.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/654405172 ]] 2023-10-14T00:23:51.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/654405172 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\6\5\4\4\0\5\1\7\2 ]] 2023-10-14T00:23:51.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.018+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.018+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.018 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.018+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.018+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cd1d700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.018+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d157780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.019 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.018+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d056800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.020 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.022+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.022+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d056180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.025 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.026+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.025 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.026+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d057e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.030+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.030+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d71b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.034+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.035 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.034+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d26e080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.040 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.042+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.040 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.042+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c7f6b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.046+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.046+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66cb17180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.050 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.050+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.051 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.050+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a1c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.056 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.058+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.057 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.058+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d4a1500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.069 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.070+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.070+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d26ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.082+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.081 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.082+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d851300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.088 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.090+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.088 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.090+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d7f2b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.107 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.106+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.107 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.106+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d7f3980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.112 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.114+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.112 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.114+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d851180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.116 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.118+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.116 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.118+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d850f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.121 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.122+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.121 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.122+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d851900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.125 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.126+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.126 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.126+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d851b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.130 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.130+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:00.131 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:00.130+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d851d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:01.143 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:01.145 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:24:01.145 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:01.151 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:01.977 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:24:01.977 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:24:01.978 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243041 2023-10-14T00:24:01.978 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:01.978 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/654405172 ]] 2023-10-14T00:24:01.978 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/654405172 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\6\5\4\4\0\5\1\7\2 ]] 2023-10-14T00:24:01.978 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:02.179 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:24:02.180 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:24:02.180 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:24:02.180 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:24:02.180 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:24:02.181 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:24:02.181 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:24:02.181 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:24:02.181 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:24:02.181 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:24:02.181 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:24:02.182 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:24:02.182 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:02.182+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:24:11.986 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:11.989 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:24:11.989 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:12.002 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:12.891 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:24:12.891 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:24:12.892 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243052 2023-10-14T00:24:12.892 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:12.892 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/654405172 ]] 2023-10-14T00:24:12.892 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/654405172 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\6\5\4\4\0\5\1\7\2 ]] 2023-10-14T00:24:12.893 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:22.897 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:24:22.898 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:22.898 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:22.898 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:23.636 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/654405172 2023-10-14T00:24:23.637 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:24:23.637 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243063 2023-10-14T00:24:23.637 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:23.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/654405172 ]] 2023-10-14T00:24:23.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/654405172 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\6\5\4\4\0\5\1\7\2 ]] 2023-10-14T00:24:23.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:24.244 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:24.246+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:24:24.245 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:24.246+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:24:24.246 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:24.246+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:24:24.249 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:24.250+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:24:28.901 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:24:28.904 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:24:28.904 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:24:28.904 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:24:28.904 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:24:28.904 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:24:28.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:24:28.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:24:28.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:24:28.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:24:28.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:24:28.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:24:28.906 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:28.902+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:24:33.645 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:33.651 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:33.652 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:33.660 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:24:34.674 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:24:34.674 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:24:34.674 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243074 2023-10-14T00:24:34.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:34.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3712341576 ]] 2023-10-14T00:24:34.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3712341576 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\6\5\4\4\0\5\1\7\2 ]] 2023-10-14T00:24:34.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/3712341576 2023-10-14T00:24:37.159 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/3712341576 until 2023-10-14T01:24:36.368678+0000 (3600 sec) 2023-10-14T00:24:37.200 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:24:37.203 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/3712341576 2023-10-14T00:24:37.922 INFO:tasks.workunit.client.0.smithi105.stderr:listed 17 entries 2023-10-14T00:24:37.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:24:37.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:38.757 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:38.758+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:24:38.758 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:38.758+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:24:38.759 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:38.758+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:24:47.955 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:24:47.955 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:47.965 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:47.976 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:48.883 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:24:48.884 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:24:48.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243088 2023-10-14T00:24:48.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:48.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3712341576 ]] 2023-10-14T00:24:48.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3712341576 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\7\1\2\3\4\1\5\7\6 ]] 2023-10-14T00:24:48.885 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:50.611 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.605+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:24:50.612 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.605+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:24:50.619 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.609+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:24:50.620 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:24:50.620 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:24:50.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:24:50.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:24:50.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:24:50.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:24:50.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:24:50.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:24:50.621+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:24:58.888 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:24:58.888 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:24:58.889 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:24:58.914 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:24:59.421 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:24:59.421+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:24:59.860 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:24:59.860 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:24:59.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243099 2023-10-14T00:24:59.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:24:59.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3712341576 ]] 2023-10-14T00:24:59.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3712341576 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\7\1\2\3\4\1\5\7\6 ]] 2023-10-14T00:24:59.862 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:24:59.998 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:59.997+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:24:59.998 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:24:59.997+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d382500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.001+0000 7f93c57a7640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.001+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c7f6a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.005 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.005+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.005 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.005+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7f3780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.017+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.016 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.017+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66c7f6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.021+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.021 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.021+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d71a380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.026 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.025+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.026 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.025+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d850f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.031 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.033+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.031 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.033+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d851f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.036 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.037+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.036 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.037+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d851e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.041 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.041+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.041 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.041+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cc180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.045+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.046 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.045+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cc880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.053+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.052 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.053+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cc700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.065 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.065+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.065 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.065+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cc500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.076 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.077+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.077 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.077+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cc980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.081+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.082 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.081+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7ccb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.089 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.089+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.089 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.089+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7ccd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.099 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.101+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.100 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.101+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7ccf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.111 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.109+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.111 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.113+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cd180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.119 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.117+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.119 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.117+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cd380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.126 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.125+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.126 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.125+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cd580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.137 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.137+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:00.137 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:25:00.137+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cd780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:25:09.692 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:09.689+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:25:09.693 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:09.689+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:25:09.693 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:09.693+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:25:09.863 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:25:09.863 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:09.863 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:09.886 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:10.768 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:25:10.768 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:25:10.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243110 2023-10-14T00:25:10.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:10.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3712341576 ]] 2023-10-14T00:25:10.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3712341576 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\7\1\2\3\4\1\5\7\6 ]] 2023-10-14T00:25:10.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:25:20.771 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:25:20.771 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:20.772 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:20.772 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:21.709 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:25:21.710 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:25:21.710 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243121 2023-10-14T00:25:21.711 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:21.711 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3712341576 ]] 2023-10-14T00:25:21.711 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3712341576 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\7\1\2\3\4\1\5\7\6 ]] 2023-10-14T00:25:21.711 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:25:31.715 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:25:31.716 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:31.716 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:31.722 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:32.736 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/3712341576 2023-10-14T00:25:32.737 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:25:32.737 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243132 2023-10-14T00:25:32.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:32.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/3712341576 ]] 2023-10-14T00:25:32.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/3712341576 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\7\1\2\3\4\1\5\7\6 ]] 2023-10-14T00:25:32.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:25:37.398 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.397+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:25:37.401 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:25:37.401 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:25:37.401 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:25:37.401 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:25:37.401 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:25:37.402 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:25:37.402 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:25:37.402 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:37.401+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:25:42.742 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:25:42.743 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:42.752 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:42.763 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:43.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:25:43.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:25:43.790 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243143 2023-10-14T00:25:43.791 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:43.791 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:25:43.791 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\3\7\1\2\3\4\1\5\7\6 ]] 2023-10-14T00:25:43.791 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist add 172.21.15.105:0/1796457399 2023-10-14T00:25:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:blocklisting 172.21.15.105:0/1796457399 until 2023-10-14T01:25:44.692704+0000 (3600 sec) 2023-10-14T00:25:45.538 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep -q 172.21.15.105:0/1796457399 2023-10-14T00:25:45.548 INFO:tasks.workunit.client.0.smithi105.stderr:+ ceph osd blocklist ls 2023-10-14T00:25:46.265 INFO:tasks.workunit.client.0.smithi105.stderr:listed 18 entries 2023-10-14T00:25:46.313 INFO:tasks.workunit.client.0.smithi105.stderr:+ PREV_CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:25:46.313 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:25:47.152 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.145+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:25:47.152 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.149+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:25:47.154 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.149+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:25:47.154 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.149+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:25:47.154 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:25:47.155 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:25:47.155 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:25:47.155 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:25:47.155 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:25:47.155 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:25:47.156 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:25:47.156 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:47.153+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:25:50.564 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:50.564+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:25:52.245 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:52.248+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:25:52.245 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:52.248+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:25:52.246 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:52.248+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:25:52.247 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:25:52.252+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:25:56.314 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:25:56.315 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:25:56.327 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:25:56.351 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:25:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:25:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:25:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243157 2023-10-14T00:25:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:25:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:25:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:25:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:26:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.008+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.004 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.008+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66d057b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.012+0000 7f93c47a5640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.009 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.012+0000 7f93c47a5640 -1 librbd::ImageState: 0x55a66c4f5900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.016+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.015 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.016+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cde80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.022 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.024+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.023 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.024+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66cd1c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.029 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.032+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.030 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.032+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cd100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.036 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.040+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.036 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.040+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cdb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.044+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.039 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.044+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d7cd980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.043 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.048+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.043 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.048+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d5fe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.049 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.052+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.049 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.052+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d5fe280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.054 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.056+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.055 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.056+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d5fe480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.060 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.064+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.061 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.064+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d5fe680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.066 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.068+0000 7f93c4fa6640 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:00.066 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T00:26:00.068+0000 7f93c4fa6640 -1 librbd::ImageState: 0x55a66d5fee00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-14T00:26:07.202 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:07.215 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:07.216 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:26:07.218 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:08.078 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:26:08.078 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:26:08.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243168 2023-10-14T00:26:08.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:08.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:26:08.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:26:08.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:26:18.081 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:18.082 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:18.085 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:26:18.085 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:18.929 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:26:18.930 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:26:18.930 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243178 2023-10-14T00:26:18.930 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:18.931 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:26:18.931 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:26:18.931 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:26:28.932 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:26:28.934 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:28.935 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:28.940 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:30.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:26:30.051 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:26:30.052 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243190 2023-10-14T00:26:30.052 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:30.052 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:26:30.053 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:26:30.053 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:26:40.068 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:40.069 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:26:40.071 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:40.075 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:41.080 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:26:41.080 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:26:41.081 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243201 2023-10-14T00:26:41.082 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:41.082 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:26:41.082 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:26:41.082 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:26:42.944 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:26:42.947+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:26:42.951 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:26:42.947+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:26:51.090 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:26:51.092 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:26:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:26:51.105 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:26:52.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:26:52.079 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:26:52.080 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243212 2023-10-14T00:26:52.080 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:26:52.080 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:26:52.081 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:26:52.081 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:27:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:27:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:02.099 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:03.148 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:27:03.149 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:27:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243223 2023-10-14T00:27:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:27:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:27:03.151 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:27:03.325 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:27:03.325 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:27:03.325 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:27:03.325 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:27:03.326 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:27:03.326 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:27:03.326 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:27:03.326 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:27:03.326 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:27:03.327 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:27:03.327 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:27:03.327 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:27:03.327 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:27:03.322+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:27:13.159 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:27:13.160 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:13.162 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:14.098 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:27:14.099 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:27:14.099 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243234 2023-10-14T00:27:14.100 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:14.100 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:27:14.100 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:27:14.100 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:27:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:27:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:25.101 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:27:25.108 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:27:25.109 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243245 2023-10-14T00:27:25.110 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:25.110 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:27:25.110 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:27:25.110 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:27:35.113 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:27:35.114 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:35.114 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:36.004 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:27:36.005 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:27:36.005 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243256 2023-10-14T00:27:36.005 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:36.006 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:27:36.006 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:27:36.006 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:27:46.007 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:46.010 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:27:46.013 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:46.028 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:46.915 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:27:46.915 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:27:46.916 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243266 2023-10-14T00:27:46.916 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:46.916 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:27:46.916 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:27:46.917 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:27:56.918 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:27:56.918 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:27:56.918 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:27:56.920 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:27:57.847 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:27:57.847 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:27:57.848 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243277 2023-10-14T00:27:57.849 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:27:57.849 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:27:57.849 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:27:57.850 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:28:05.737 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:05.733+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:28:05.737 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:05.733+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:28:05.738 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:05.737+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:28:07.851 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:07.853 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:28:07.859 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:07.876 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:08.753 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:28:08.753 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:28:08.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243288 2023-10-14T00:28:08.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:08.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:28:08.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:28:08.755 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:28:10.280 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:28:10.281+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:28:10.281 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:28:10.281+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:28:10.281 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:28:10.281+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:28:16.233 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.232+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:28:16.233 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.232+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:28:16.233 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.232+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:28:16.234 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.232+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:28:16.234 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.232+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:28:16.234 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.232+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:28:16.580 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:28:16.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:28:16.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:28:16.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:28:16.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:28:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:28:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:28:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:28:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.580+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:28:16.583 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.584+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:28:16.583 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.584+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:28:16.583 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:16.584+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:28:18.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:18.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:18.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:28:18.800 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:19.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:28:19.770 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:28:19.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243299 2023-10-14T00:28:19.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:19.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:28:19.772 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:28:19.772 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:28:23.462 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:23.456+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:28:23.463 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:23.460+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:28:23.464 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:23.460+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:28:23.467 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:23.468+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:28:27.605 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:27.604+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:28:27.614 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:28:27.608+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:28:29.777 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:28:29.778 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:29.779 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:29.800 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:30.800 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:28:30.801 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:28:30.807 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243310 2023-10-14T00:28:30.807 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:30.807 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:28:30.807 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:28:30.808 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:28:40.810 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:28:40.810 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:40.814 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:40.822 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:41.794 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:28:41.795 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:28:41.795 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243321 2023-10-14T00:28:41.795 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:41.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:28:41.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:28:41.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:28:51.468 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:28:51.468+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:28:51.797 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:28:51.800 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:28:51.801 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:28:51.822 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:28:52.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:28:52.787 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:28:52.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243332 2023-10-14T00:28:52.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:28:52.789 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:28:52.789 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:28:52.789 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:28:58.560 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:28:58.559+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:29:02.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:02.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:02.804 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:02.804 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:29:03.742 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:29:03.748 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:29:03.749 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243343 2023-10-14T00:29:03.749 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:03.749 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:29:03.750 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:29:03.750 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:29:13.754 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:29:13.755 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:13.756 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:13.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:14.637 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:29:14.637 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:29:14.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243354 2023-10-14T00:29:14.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:14.639 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:29:14.639 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:29:14.639 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:29:18.867 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:29:18.867+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:29:24.640 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:24.641 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:24.645 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:29:24.660 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:25.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:29:25.699 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:29:25.700 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243365 2023-10-14T00:29:25.700 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:25.700 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:29:25.700 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:29:25.701 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:29:35.709 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:29:35.709 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:35.710 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:35.745 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:36.679 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:29:36.680 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:29:36.680 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243376 2023-10-14T00:29:36.680 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:36.681 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:29:36.681 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:29:36.681 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:29:46.682 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:29:46.691 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:46.704 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:46.705 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:47.595 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:29:47.596 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:29:47.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243387 2023-10-14T00:29:47.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:47.597 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:29:47.597 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:29:47.597 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:29:57.598 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:29:57.599 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:29:57.604 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:29:57.604 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:29:58.523 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:29:58.523 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:29:58.523 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243398 2023-10-14T00:29:58.524 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:29:58.524 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:29:58.524 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:29:58.524 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:30:00.080 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:30:00.078+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:30:00.081 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:30:00.078+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:30:00.081 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:30:00.078+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:30:08.533 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:08.533 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:30:08.535 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:08.547 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:09.467 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:30:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:30:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243409 2023-10-14T00:30:09.471 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:09.471 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:30:09.471 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:30:09.471 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:30:19.472 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:30:19.473 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:19.473 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:19.473 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:20.416 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:30:20.433 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:30:20.434 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243420 2023-10-14T00:30:20.434 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:20.434 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:30:20.434 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:30:20.435 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:30:30.436 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:30:30.436 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:30.437 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:30.440 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:31.265 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:30:31.265 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:30:31.266 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243431 2023-10-14T00:30:31.266 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:31.266 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:30:31.266 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:30:31.267 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:30:36.890 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:30:36.889+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:30:36.891 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:30:36.889+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:30:41.268 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:41.269 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:41.270 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:41.290 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:30:42.147 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:30:42.148 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:30:42.148 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243442 2023-10-14T00:30:42.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:42.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:30:42.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:30:42.149 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:30:52.150 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:30:52.152 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:30:52.160 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:30:52.164 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:30:53.205 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:30:53.206 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:30:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243453 2023-10-14T00:30:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:30:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:30:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:30:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:31:03.211 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:31:03.215 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:03.215 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:04.121 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:31:04.122 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:31:04.127 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243464 2023-10-14T00:31:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:31:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:31:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:31:14.130 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:14.132 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:31:14.135 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:14.138 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:15.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:31:15.060 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:31:15.061 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243475 2023-10-14T00:31:15.061 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:15.061 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:31:15.062 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:31:15.062 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:31:25.068 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:31:25.069 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:25.069 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:25.069 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:25.956 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:31:25.957 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:31:25.958 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243485 2023-10-14T00:31:25.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:25.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:31:25.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:31:25.959 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:31:31.225 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:31:31.224+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:31:31.225 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:31:31.224+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:31:35.552 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:35.552+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:31:35.553 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:35.552+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:31:35.554 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:35.556+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:31:35.555 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:35.556+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:31:35.555 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:35.556+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:31:35.555 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:35.556+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:31:35.960 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:35.965 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:31:35.975 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:35.981 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:36.872 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:31:36.872 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:31:36.873 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243496 2023-10-14T00:31:36.873 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:36.873 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:31:36.873 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:31:36.874 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:31:43.791 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.792+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:31:43.796 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.792+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:31:43.797 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.792+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:31:43.803 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:31:43.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:31:43.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:31:43.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:31:43.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:31:43.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:31:43.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:31:43.805 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:31:43.805 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:31:43.804+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:31:46.881 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:46.887 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:46.887 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:31:46.890 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:47.864 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:31:47.864 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:31:47.866 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243507 2023-10-14T00:31:47.866 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:47.866 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:31:47.866 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:31:47.867 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:31:57.872 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:31:57.873 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:31:57.873 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:31:57.881 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:31:58.752 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:31:58.752 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:31:58.753 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243518 2023-10-14T00:31:58.753 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:31:58.753 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:31:58.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:31:58.754 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:32:08.757 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:08.761 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:08.766 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:32:08.774 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:09.672 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:32:09.672 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:32:09.679 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243529 2023-10-14T00:32:09.679 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:09.680 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:32:09.680 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:32:09.680 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:32:14.173 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:14.167+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:32:14.175 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:14.167+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:32:19.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:32:19.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:19.705 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:19.709 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:20.717 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:32:20.717 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:32:20.717 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243540 2023-10-14T00:32:20.718 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:20.718 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:32:20.718 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:32:20.718 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:32:25.048 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:32:25.049 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:32:25.049 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:32:25.049 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:32:25.049 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:32:25.050 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:32:25.050 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:32:25.050 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:32:25.050 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:32:25.050 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:32:25.051 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:32:25.051 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:32:25.051 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:32:25.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:32:30.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:32:30.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:30.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:30.739 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:32:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:32:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243551 2023-10-14T00:32:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:32:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:32:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:32:41.577 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:32:41.578 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:41.579 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:41.605 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:42.437 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:32:42.438 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:32:42.438 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243562 2023-10-14T00:32:42.438 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:42.439 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:32:42.439 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:32:42.439 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:32:52.440 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:32:52.441 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:32:52.448 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:32:52.456 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:32:53.463 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:32:53.464 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:32:53.465 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243573 2023-10-14T00:32:53.465 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:32:53.466 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:32:53.466 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:32:53.466 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:33:03.468 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:33:03.471 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:03.473 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:03.475 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:04.485 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:33:04.491 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:33:04.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243584 2023-10-14T00:33:04.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:04.492 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:33:04.493 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:33:04.493 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:33:14.496 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:14.498 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:14.499 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:33:14.500 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:15.428 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:33:15.428 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:33:15.429 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243595 2023-10-14T00:33:15.429 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:15.429 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:33:15.429 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:33:15.430 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:33:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:33:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:25.443 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:25.446 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:26.447 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:33:26.453 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:33:26.455 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243606 2023-10-14T00:33:26.455 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:26.455 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:33:26.456 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:33:26.456 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:33:36.466 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:33:36.467 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:36.467 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:36.475 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:37.344 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:33:37.344 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:33:37.346 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243617 2023-10-14T00:33:37.346 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:37.347 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:33:37.347 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:33:37.347 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:33:47.348 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:33:47.349 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:47.355 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:47.362 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:48.191 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:33:48.192 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:33:48.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243628 2023-10-14T00:33:48.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:48.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:33:48.194 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:33:48.194 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:33:58.194 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:33:58.195 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:33:58.202 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:33:58.204 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:33:59.146 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:33:59.146 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:33:59.147 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243639 2023-10-14T00:33:59.147 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:33:59.147 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:33:59.147 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:33:59.148 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:34:04.916 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:34:04.911+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:34:09.159 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:34:09.160 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:09.160 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:09.161 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:10.133 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:34:10.133 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:34:10.134 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243650 2023-10-14T00:34:10.134 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:10.134 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:34:10.134 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:34:10.135 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:34:16.620 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:34:16.615+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:34:16.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:34:16.631+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:34:20.142 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:20.142 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:34:20.151 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:20.153 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:21.137 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:34:21.137 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:34:21.138 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243661 2023-10-14T00:34:21.138 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:21.138 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:34:21.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:34:21.139 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:34:31.140 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:34:31.140 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:31.140 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:31.146 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:32.131 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:34:32.131 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:34:32.132 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243672 2023-10-14T00:34:32.132 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:32.132 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:34:32.132 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:34:32.132 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:34:42.133 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:34:42.134 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:42.137 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:42.138 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:42.782 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:34:42.782 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:34:42.783 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243682 2023-10-14T00:34:42.783 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:42.783 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:34:42.783 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:34:42.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:34:52.784 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:34:52.785 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:34:52.785 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:34:52.801 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:34:53.664 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:34:53.664 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:34:53.666 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243693 2023-10-14T00:34:53.666 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:34:53.667 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:34:53.667 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:34:53.667 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:35:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:03.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:35:03.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:04.616 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:35:04.619 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:35:04.620 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243704 2023-10-14T00:35:04.620 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:04.620 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:35:04.620 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:35:04.621 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:35:10.292 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:35:10.290+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:35:10.292 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:35:10.290+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:35:10.293 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:35:10.290+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:35:14.621 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:35:14.632 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:14.635 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:14.639 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:15.505 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:35:15.506 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:35:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243715 2023-10-14T00:35:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:15.508 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:35:15.508 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:35:15.508 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:35:25.509 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:25.510 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:25.510 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:35:25.515 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:26.501 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:35:26.501 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:35:26.504 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243726 2023-10-14T00:35:26.504 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:26.504 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:35:26.504 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:35:26.504 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:35:36.509 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:36.509 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:36.510 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:35:36.515 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:37.418 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:37.413+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:35:37.419 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:37.413+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:35:37.419 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:37.413+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:35:37.419 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:37.413+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:35:37.463 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:35:37.464 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:35:37.465 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243737 2023-10-14T00:35:37.465 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:37.465 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:35:37.465 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:35:37.466 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:35:41.907 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.901+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:35:41.915 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.909+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:35:41.917 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.913+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:35:41.917 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.913+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:35:41.921 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.917+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:35:41.925 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.917+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:35:41.925 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:35:41.917+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:35:47.467 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:47.467 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:47.475 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:47.475 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:35:48.308 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:35:48.308 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:35:48.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243748 2023-10-14T00:35:48.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:48.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:35:48.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:35:48.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:35:58.311 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:35:58.311 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:35:58.319 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:35:58.321 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:35:59.194 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:35:59.194 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:35:59.194 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243759 2023-10-14T00:35:59.195 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:35:59.195 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:35:59.195 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:35:59.195 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:36:09.197 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:09.197 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:09.198 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:09.203 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:36:10.152 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:36:10.152 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243770 2023-10-14T00:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:36:20.160 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:20.161 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:36:20.170 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:20.177 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:21.140 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:36:21.141 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:36:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243781 2023-10-14T00:36:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:36:21.143 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:36:21.143 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:36:31.162 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:36:31.163 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:31.167 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:31.171 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:32.276 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:36:32.277 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:36:32.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243792 2023-10-14T00:36:32.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:32.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:36:32.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:36:32.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:36:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:42.283 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:42.289 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:36:42.291 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:43.182 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:36:43.183 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:36:43.183 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243803 2023-10-14T00:36:43.184 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:43.184 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:36:43.184 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:36:43.184 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:36:48.823 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:36:48.819+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:36:48.831 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:36:48.823+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:36:53.186 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:36:53.187 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:36:53.187 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:36:53.192 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:36:54.142 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:36:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:36:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243814 2023-10-14T00:36:54.144 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:36:54.144 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:36:54.144 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:36:54.144 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:36:59.221 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:36:59.219+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:37:04.035 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:37:04.031+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:37:04.036 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:37:04.031+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:37:04.053 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:37:04.051+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:37:04.146 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:04.146 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:37:04.149 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:04.163 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:05.367 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:37:05.371 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:37:05.372 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243825 2023-10-14T00:37:05.373 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:05.373 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:37:05.373 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:37:05.373 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:37:06.347 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:37:06.343+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:37:06.347 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:37:06.343+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:37:06.360 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:37:06.359+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:37:15.381 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:37:15.383 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:15.384 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:15.386 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:16.210 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:37:16.211 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:37:16.211 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243836 2023-10-14T00:37:16.211 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:16.212 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:37:16.212 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:37:16.212 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:37:26.230 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:26.233 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:26.234 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:37:26.240 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:27.270 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:37:27.271 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:37:27.272 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243847 2023-10-14T00:37:27.272 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:27.272 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:37:27.273 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:37:27.273 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:37:37.274 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:37:37.275 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:37.275 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:37.290 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:38.277 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:37:38.278 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:37:38.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243858 2023-10-14T00:37:38.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:38.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:37:38.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:37:38.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:37:48.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:48.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:48.282 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:37:48.298 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:37:49.232 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:37:49.232 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:37:49.233 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243869 2023-10-14T00:37:49.233 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:37:49.233 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:37:49.233 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:37:49.233 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:37:59.240 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:37:59.245 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:37:59.246 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:37:59.263 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:00.245 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:38:00.245 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:38:00.246 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243880 2023-10-14T00:38:00.247 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:00.247 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:38:00.247 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:38:00.247 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:38:10.264 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:10.267 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:10.271 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:10.272 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:38:11.171 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:38:11.172 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:38:11.173 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243891 2023-10-14T00:38:11.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:11.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:38:11.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:38:11.175 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:38:18.839 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:38:18.834+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:38:18.842 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:38:18.838+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:38:18.842 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:38:18.838+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:38:18.863 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:38:18.850+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:38:18.863 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:38:18.850+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:38:21.176 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:21.178 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:21.181 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:38:21.185 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:22.215 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:38:22.218 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:38:22.219 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243902 2023-10-14T00:38:22.219 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:22.220 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:38:22.220 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:38:22.220 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:38:32.224 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:32.225 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:32.227 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:32.228 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:38:33.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:38:33.059 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:38:33.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243913 2023-10-14T00:38:33.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:33.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:38:33.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:38:33.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:38:43.064 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:43.064 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:43.065 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:38:43.085 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:43.954 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:38:43.954 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:38:43.956 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243923 2023-10-14T00:38:43.956 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:43.957 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:38:43.957 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:38:43.957 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:38:53.958 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:38:53.959 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:38:53.960 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:38:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:38:54.968 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:38:54.969 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:38:54.969 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243934 2023-10-14T00:38:54.969 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:38:54.970 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:38:54.970 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:38:54.970 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:39:04.063 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:04.057+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:39:04.064 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:04.061+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:39:04.065 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:04.061+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:39:04.065 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:04.061+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:39:04.066 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:04.061+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:39:04.971 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:39:04.973 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:04.973 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:04.973 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:05.191 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:39:05.185+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:39:05.191 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:39:05.185+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:39:05.191 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:39:05.185+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:39:05.929 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:39:05.929 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:39:05.930 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243945 2023-10-14T00:39:05.930 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:05.930 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:39:05.931 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:39:05.931 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:39:15.932 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:39:15.932 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:15.932 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:15.936 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:17.045 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:39:17.045 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:39:17.046 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243957 2023-10-14T00:39:17.046 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:17.046 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:39:17.047 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:39:17.047 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:39:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:39:27.056 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:27.066 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:27.068 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:28.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:39:28.058 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:39:28.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243968 2023-10-14T00:39:28.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:28.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:39:28.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:39:28.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:39:38.060 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:39:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:38.064 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:39.015 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:39:39.015 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:39:39.016 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243979 2023-10-14T00:39:39.016 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:39.016 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:39:39.016 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:39:39.016 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:39:49.028 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:39:49.033 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:39:49.050 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:39:49.057 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:39:50.076 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:39:50.077 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:39:50.078 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697243990 2023-10-14T00:39:50.078 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:39:50.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:39:50.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:39:50.079 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:39:57.629 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:39:57.630 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:39:57.630 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:39:57.630 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:39:57.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:39:57.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:39:57.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:39:57.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:39:57.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:39:57.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:39:57.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:39:57.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:39:57.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:57.628+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:39:59.619 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:59.616+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:39:59.620 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:59.616+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:39:59.620 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:39:59.616+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:40:00.084 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:00.086 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:40:00.097 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:00.098 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:00.967 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:40:00.967 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:40:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244000 2023-10-14T00:40:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:40:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:40:00.969 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:40:10.972 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:10.973 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:10.979 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:10.979 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:40:11.971 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:40:11.972 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:40:11.972 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244011 2023-10-14T00:40:11.973 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:11.973 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:40:11.973 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:40:11.973 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:40:13.587 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:13.583+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:40:13.590 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:13.587+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:40:13.590 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:13.587+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:40:13.590 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:13.587+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:40:21.974 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:40:21.975 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:21.979 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:21.985 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:22.833 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:40:22.833 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:40:22.833 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244022 2023-10-14T00:40:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:40:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:40:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:40:27.615 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:27.611+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:40:27.618 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:27.615+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:40:27.625 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:27.623+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:40:32.837 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:40:32.838 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:32.838 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:32.850 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:33.761 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:40:33.762 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:40:33.762 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244033 2023-10-14T00:40:33.762 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:33.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:40:33.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:40:33.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:40:37.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:40:37.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:40:37.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:40:37.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:40:37.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:40:37.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:40:37.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:40:37.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:40:37.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:40:37.890 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:40:37.890 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:40:37.890 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:40:37.891 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:40:37.883+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:40:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:40:43.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:44.613 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:40:44.613 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:40:44.614 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244044 2023-10-14T00:40:44.614 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:44.614 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:40:44.615 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:40:44.615 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:40:54.622 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:40:54.627 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:40:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:40:54.642 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:40:55.546 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:40:55.547 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:40:55.548 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244055 2023-10-14T00:40:55.548 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:40:55.548 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:40:55.549 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:40:55.549 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:41:05.550 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:41:05.563 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:05.566 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:05.575 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:06.373 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:41:06.376 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:41:06.377 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244066 2023-10-14T00:41:06.378 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:06.378 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:41:06.378 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:41:06.378 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:41:14.820 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.814+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:41:14.821 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.814+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:41:14.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:41:14.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:41:14.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:41:14.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:41:14.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:41:14.825 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:41:14.825 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:41:14.825 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:41:14.825 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:41:14.818+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:41:16.384 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:16.386 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:16.387 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:16.396 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:41:17.319 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:41:17.319 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:41:17.320 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244077 2023-10-14T00:41:17.320 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:17.320 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:41:17.320 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:41:17.321 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:41:27.321 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:27.322 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:27.323 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:27.331 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:41:28.296 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:41:28.307 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:41:28.308 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244088 2023-10-14T00:41:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:41:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:41:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:41:38.312 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:38.320 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:41:38.320 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:38.320 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:39.282 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:41:39.286 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:41:39.291 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244099 2023-10-14T00:41:39.291 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:41:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:41:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:41:49.292 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:41:49.293 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:41:49.298 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:41:49.313 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:41:50.241 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:41:50.242 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:41:50.242 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244110 2023-10-14T00:41:50.243 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:41:50.243 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:41:50.243 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:41:50.243 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:41:56.826 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:41:56.822+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:42:00.245 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:42:00.245 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:00.245 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:00.268 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:01.249 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:42:01.249 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:42:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244121 2023-10-14T00:42:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:42:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:42:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:42:11.253 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:42:11.253 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:11.254 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:11.267 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:12.164 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:42:12.165 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:42:12.165 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244132 2023-10-14T00:42:12.165 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:12.166 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:42:12.166 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:42:12.166 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:42:22.171 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:22.172 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:42:22.182 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:23.125 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:42:23.127 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:42:23.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244143 2023-10-14T00:42:23.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:23.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:42:23.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:42:23.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:42:32.839 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:42:32.833+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:42:32.840 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:42:32.833+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:42:32.840 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:42:32.833+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:42:33.130 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:33.131 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:33.141 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:42:33.142 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:34.056 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:42:34.057 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:42:34.057 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244154 2023-10-14T00:42:34.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:34.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:42:34.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:42:34.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:42:44.059 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:42:44.059 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:44.060 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:44.060 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:44.981 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:42:44.999 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:42:45.000 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244165 2023-10-14T00:42:45.000 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:45.000 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:42:45.001 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:42:45.001 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:42:55.016 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:42:55.016 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:42:55.018 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:42:55.019 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:42:55.868 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:42:55.868 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:42:55.869 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244175 2023-10-14T00:42:55.869 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:42:55.869 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:42:55.869 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:42:55.870 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:43:01.454 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.445+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:43:01.455 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.445+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:43:01.455 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.445+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:43:01.459 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:43:01.460 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:43:01.460 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:43:01.460 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:43:01.460 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:43:01.460 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:43:01.461 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:43:01.461 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:01.453+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:43:05.871 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:43:05.871 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:05.873 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:05.876 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:06.792 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:43:06.792 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:43:06.793 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244186 2023-10-14T00:43:06.793 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:06.793 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:43:06.793 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:43:06.793 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:43:16.794 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:43:16.794 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:16.800 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:16.811 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:17.846 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:43:17.847 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:43:17.853 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244197 2023-10-14T00:43:17.854 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:17.854 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:43:17.854 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:43:17.854 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:43:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:43:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:27.859 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:28.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:43:28.756 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:43:28.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244208 2023-10-14T00:43:28.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:28.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:43:28.758 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:43:28.758 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:43:38.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:43:38.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:38.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:38.778 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:39.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:39.128+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:43:39.135 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:39.132+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:43:39.139 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:39.136+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:43:39.139 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:39.136+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:43:39.597 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:43:39.597 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:43:39.598 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244219 2023-10-14T00:43:39.599 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:39.599 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:43:39.599 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:43:39.599 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:43:40.611 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:43:40.608+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:43:42.786 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:42.784+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:43:42.786 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:42.784+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:43:42.793 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:43:42.788+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:43:49.600 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:43:49.601 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:43:49.602 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:43:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:43:50.606 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:43:50.606 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:43:50.607 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244230 2023-10-14T00:43:50.607 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:43:50.607 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:43:50.607 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:43:50.608 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:44:00.610 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:00.616 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:00.622 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:44:00.625 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:01.530 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:44:01.531 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:44:01.531 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244241 2023-10-14T00:44:01.532 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:01.532 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:44:01.532 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:44:01.532 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:44:11.533 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:44:11.533 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:11.535 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:11.555 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:12.483 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:44:12.483 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:44:12.484 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244252 2023-10-14T00:44:12.484 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:12.484 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:44:12.484 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:44:12.485 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:44:22.486 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:22.486 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:44:22.487 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:22.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:23.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:44:23.382 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:44:23.383 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244263 2023-10-14T00:44:23.383 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:23.383 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:44:23.384 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:44:23.384 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:44:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:44:33.400 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:33.401 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:33.429 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:34.420 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:44:34.420 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:44:34.421 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244274 2023-10-14T00:44:34.421 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:34.421 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:44:34.421 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:44:34.422 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:44:37.269 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:44:37.267+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:44:37.291 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:44:37.283+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:44:37.291 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:44:37.287+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:44:44.423 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:44:44.424 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:44.435 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:44.435 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:45.435 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:44:45.438 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:44:45.446 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244285 2023-10-14T00:44:45.446 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:45.446 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:44:45.447 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:44:45.447 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:44:55.459 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:44:55.459 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:44:55.463 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:44:55.483 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:44:56.308 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:44:56.309 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:44:56.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244296 2023-10-14T00:44:56.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:44:56.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:44:56.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:44:56.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:45:06.327 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:06.327 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:45:06.328 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:06.333 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:07.243 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:45:07.243 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:45:07.244 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244307 2023-10-14T00:45:07.244 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:07.244 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:45:07.244 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:45:07.244 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:45:17.251 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:17.251 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:17.251 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:45:17.256 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:18.191 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:45:18.192 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:45:18.192 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244318 2023-10-14T00:45:18.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:18.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:45:18.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:45:18.193 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:45:23.768 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:23.766+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:45:23.768 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:23.766+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:45:23.769 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:23.766+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:45:27.127 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.126+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:45:27.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.130+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:45:27.131 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.130+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:45:27.146 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.142+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:45:27.148 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:45:27.149 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:45:27.149 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:45:27.149 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:45:27.150 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:45:27.150 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:45:27.150 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:45:27.151 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:45:27.146+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:45:28.199 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:45:28.199 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:28.199 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:28.201 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:28.198+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:45:28.217 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:29.155 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:45:29.156 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:45:29.156 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244329 2023-10-14T00:45:29.157 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:29.157 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:45:29.157 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:45:29.157 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:45:38.883 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:38.878+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:45:38.883 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:38.878+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:45:38.883 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:45:38.878+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:45:39.174 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:45:39.177 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:39.179 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:39.189 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:40.168 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:45:40.168 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:45:40.169 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244340 2023-10-14T00:45:40.169 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:40.169 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:45:40.169 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:45:40.170 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:45:50.172 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:45:50.172 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:45:50.177 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:45:50.184 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:45:51.253 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:45:51.253 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:45:51.254 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244351 2023-10-14T00:45:51.254 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:45:51.254 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:45:51.255 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:45:51.255 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:01.255 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:46:01.256 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:01.259 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:01.286 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:02.172 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:46:02.173 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:46:02.173 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244362 2023-10-14T00:46:02.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:02.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:46:02.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:46:02.174 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:09.051 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:46:09.053 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:46:09.054 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:46:09.055 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:46:09.055 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:46:09.055 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:46:09.055 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:46:09.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:46:09.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:46:09.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:46:09.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:09.050+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:46:12.175 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:46:12.175 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:12.176 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:12.211 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:13.024 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:46:13.024 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:46:13.025 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244373 2023-10-14T00:46:13.025 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:13.025 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:46:13.025 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:46:13.025 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:23.032 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:23.043 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:23.047 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:46:23.048 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:23.992 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:46:23.993 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:46:23.993 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244383 2023-10-14T00:46:23.994 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:23.994 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:46:23.994 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:46:23.994 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:34.000 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:46:34.001 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:34.002 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:34.007 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:34.926 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:46:34.927 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:46:34.927 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244394 2023-10-14T00:46:34.927 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:34.928 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:46:34.928 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:46:34.928 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:44.930 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:44.933 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:44.933 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:44.940 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:46:45.701 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:45.697+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:46:45.705 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:45.701+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:46:45.705 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:45.701+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:46:45.835 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:46:45.838 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:46:45.847 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244405 2023-10-14T00:46:45.848 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:45.848 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:46:45.848 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:46:45.848 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:55.849 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:46:55.850 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:46:55.850 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:46:55.852 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:46:56.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:46:56.722 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:46:56.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244416 2023-10-14T00:46:56.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:46:56.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:46:56.724 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:46:56.724 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:46:57.218 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:57.213+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:46:57.221 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:57.217+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:46:57.221 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:46:57.217+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:46:59.282 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:46:59.277+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:46:59.283 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:46:59.277+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:46:59.283 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:46:59.277+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:47:00.390 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:00.385+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:47:00.390 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:00.385+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:47:00.393 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:00.389+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:47:06.733 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:47:06.733 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:07.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:47:07.573 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:47:07.574 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244427 2023-10-14T00:47:07.575 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:07.575 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:47:07.575 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:47:07.575 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:47:08.261 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:47:08.257+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:47:08.261 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:47:08.257+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:47:12.688 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:12.685+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:47:12.692 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:12.689+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:47:12.702 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:12.693+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:47:17.575 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:17.576 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:47:17.589 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:17.597 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:18.595 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:47:18.602 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:47:18.603 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244438 2023-10-14T00:47:18.603 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:18.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:47:18.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:47:18.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:47:23.840 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.833+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:47:23.841 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.837+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:47:23.861 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:47:23.861 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:47:23.862 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:47:23.862 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:47:23.862 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:47:23.862 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:47:23.862 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:47:23.862 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:47:23.863 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:23.857+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:47:28.605 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:47:28.607 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:28.618 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:29.571 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:47:29.571 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:47:29.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244449 2023-10-14T00:47:29.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:29.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:47:29.572 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:47:29.573 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:47:39.574 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:39.575 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:47:39.584 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:40.463 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:47:40.464 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:47:40.469 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244460 2023-10-14T00:47:40.469 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:40.469 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:47:40.470 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:47:40.470 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:47:50.476 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:47:50.485 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:47:50.490 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:47:50.507 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:47:51.352 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:47:51.353 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:47:51.354 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244471 2023-10-14T00:47:51.354 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:47:51.354 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:47:51.354 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:47:51.354 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:47:52.327 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:47:52.328+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:47:54.241 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:54.240+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:47:54.241 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:54.240+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:47:54.241 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:47:54.240+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:48:01.355 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:01.356 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:01.369 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:48:01.371 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:02.316 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:48:02.316 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:48:02.319 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244482 2023-10-14T00:48:02.319 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:02.319 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:48:02.319 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:48:02.320 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:48:03.954 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:03.952+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:48:03.955 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:03.952+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:48:03.973 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:03.964+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:48:03.973 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:03.964+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:48:03.975 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:03.976+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:48:03.976 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:03.976+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:48:08.682 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:08.680+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:48:08.683 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:08.680+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:48:12.321 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:12.321 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:12.322 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:12.336 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:48:13.208 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:48:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:48:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244493 2023-10-14T00:48:13.210 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:13.210 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:48:13.210 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:48:13.210 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:48:23.219 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:48:23.221 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:23.222 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:23.227 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:24.185 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:48:24.186 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:48:24.187 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244504 2023-10-14T00:48:24.187 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:24.187 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:48:24.187 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:48:24.187 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:48:34.188 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:48:34.190 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:34.191 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:34.197 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:35.040 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:48:35.040 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:48:35.041 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244515 2023-10-14T00:48:35.041 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:35.041 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:48:35.042 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:48:35.042 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:48:45.043 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:48:45.045 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:45.046 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:45.049 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:45.081 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:48:45.079+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:48:46.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:48:46.059 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:48:46.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244526 2023-10-14T00:48:46.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:48:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:48:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:48:53.773 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:48:53.767+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:48:53.774 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:48:53.767+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:48:56.061 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:48:56.061 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:48:56.065 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:48:56.078 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:48:56.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:48:56.983 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:48:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244536 2023-10-14T00:48:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:48:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:48:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:48:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:49:04.840 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:49:04.839+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:49:04.841 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:49:04.839+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:49:04.841 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:49:04.839+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:49:06.992 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:49:06.995 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:06.996 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:07.004 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:07.905 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:49:07.905 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:49:07.907 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244547 2023-10-14T00:49:07.907 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:07.907 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:49:07.907 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:49:07.907 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:49:17.909 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:49:17.910 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:17.915 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:17.920 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:18.859 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:49:18.860 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:49:18.860 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244558 2023-10-14T00:49:18.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:18.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:49:18.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:49:18.861 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:49:28.863 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:49:28.866 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:28.868 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:28.870 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:29.786 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:49:29.786 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:49:29.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244569 2023-10-14T00:49:29.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:29.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:49:29.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:49:29.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:49:39.790 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:39.794 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:49:39.796 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:39.798 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:40.721 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:49:40.722 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:49:40.722 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244580 2023-10-14T00:49:40.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:40.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:49:40.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:49:40.723 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:49:50.728 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:49:50.728 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:49:50.729 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:49:50.729 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:49:51.807 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:49:51.807 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:49:51.808 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244591 2023-10-14T00:49:51.808 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:49:51.808 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:49:51.808 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:49:51.809 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:50:01.810 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:50:01.811 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:01.813 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:01.813 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:02.786 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:50:02.786 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:50:02.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244602 2023-10-14T00:50:02.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:02.787 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:50:02.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:50:02.788 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:50:12.806 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:50:12.810 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:12.812 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:12.819 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:13.543 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:50:13.542+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:50:13.543 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:50:13.542+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:50:13.543 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:50:13.542+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:50:13.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:50:13.770 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:50:13.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244613 2023-10-14T00:50:13.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:13.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:50:13.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:50:13.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:50:23.712 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:50:23.712 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:50:23.712 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:50:23.712 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:50:23.713 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:50:23.713 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:50:23.713 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:50:23.713 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:23.709+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:50:23.772 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:50:23.772 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:23.774 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:23.775 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:24.696 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:50:24.696 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:50:24.697 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244624 2023-10-14T00:50:24.697 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:24.697 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:50:24.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:50:24.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:50:33.264 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:33.261+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:50:33.265 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:33.261+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:50:33.275 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:33.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:50:33.275 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:33.269+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:50:33.286 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:50:33.281+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:50:34.699 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:50:34.699 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:34.702 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:34.702 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:35.651 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:50:35.652 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:50:35.652 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244635 2023-10-14T00:50:35.653 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:35.653 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:50:35.653 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:50:35.653 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:50:45.668 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:50:45.671 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:45.671 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:45.673 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:46.762 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:50:46.762 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:50:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244646 2023-10-14T00:50:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:50:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:50:46.764 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:50:51.997 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:50:51.993+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:50:51.998 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:50:51.993+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:50:51.998 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:50:51.993+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:50:56.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:50:56.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:50:56.769 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:50:56.776 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:50:57.674 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:50:57.674 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:50:57.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244657 2023-10-14T00:50:57.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:50:57.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:50:57.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:50:57.675 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:51:07.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:51:07.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:07.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:07.683 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:08.583 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:51:08.583 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:51:08.591 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244668 2023-10-14T00:51:08.591 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:08.591 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:51:08.591 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:51:08.591 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:51:13.417 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:51:13.417+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:51:13.420 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:51:13.421+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:51:13.420 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:51:13.421+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:51:13.425 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:51:13.425+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:51:18.592 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:51:18.593 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:18.606 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:18.610 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:51:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:51:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244679 2023-10-14T00:51:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:51:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:51:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:51:29.487 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:51:29.488 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:29.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:29.502 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:30.398 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:51:30.399 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:51:30.399 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244690 2023-10-14T00:51:30.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:30.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:51:30.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:51:30.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:51:40.401 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:51:40.405 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:41.398 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:51:41.399 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:51:41.399 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244701 2023-10-14T00:51:41.399 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:41.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:51:41.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:51:41.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:51:51.401 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:51:51.402 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:51:51.414 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:51:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:51:52.334 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:51:52.339 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:51:52.342 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244712 2023-10-14T00:51:52.342 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:51:52.342 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:51:52.342 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:51:52.343 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:52:02.345 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:52:02.346 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:02.346 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:02.350 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:03.305 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:52:03.305 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:52:03.306 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244723 2023-10-14T00:52:03.306 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:03.306 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:52:03.306 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:52:03.306 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:52:12.010 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:12.007+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:52:12.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:12.011+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:52:12.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:12.011+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:52:12.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:12.011+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:52:13.307 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:52:13.315 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:13.335 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:13.335 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:14.279 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:52:14.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:52:14.280 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244734 2023-10-14T00:52:14.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:14.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:52:14.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:52:14.281 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:52:14.381 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:52:14.379+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:52:14.381 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:52:14.379+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:52:24.286 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:52:24.287 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:24.290 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:24.296 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:25.228 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:52:25.229 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:52:25.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244745 2023-10-14T00:52:25.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:25.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:52:25.239 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:52:25.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:52:35.257 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:35.258 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:35.258 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:52:35.272 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:36.380 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:52:36.380 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:52:36.380 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244756 2023-10-14T00:52:36.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:36.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:52:36.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:52:36.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:52:46.386 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:52:46.387 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:46.389 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:46.409 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:46.964 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.963+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:52:46.970 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.971+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:52:46.970 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.971+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:52:46.976 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:52:46.976 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:52:46.976 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:52:46.976 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:52:46.976 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:52:46.977 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:52:46.977 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:52:46.977 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:52:46.977 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:46.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:52:47.337 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:52:47.338 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:52:47.340 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244767 2023-10-14T00:52:47.340 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:47.340 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:52:47.340 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:52:47.341 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:52:51.273 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:52:51.271+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:52:57.342 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:52:57.342 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:52:57.342 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:52:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:52:58.294 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:52:58.296 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:52:58.297 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244778 2023-10-14T00:52:58.297 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:52:58.297 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:52:58.298 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:52:58.298 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:53:00.793 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:00.791+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:53:00.800 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:00.799+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:53:00.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:00.803+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:53:00.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:00.803+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:53:00.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:00.807+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:53:00.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:00.807+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:53:04.473 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:04.471+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:53:08.300 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:53:08.300 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:08.300 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:08.301 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:09.125 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:53:09.126 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:53:09.126 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244789 2023-10-14T00:53:09.126 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:09.127 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:53:09.127 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:53:09.127 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:53:14.443 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:14.430+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:53:14.452 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:14.450+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:53:14.452 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:14.450+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:53:14.462 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:14.458+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:53:19.138 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:53:19.139 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:19.144 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:19.147 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:20.049 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:53:20.049 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:53:20.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244800 2023-10-14T00:53:20.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:20.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:53:20.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:53:20.050 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:53:26.083 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:53:26.082+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:53:26.083 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:53:26.082+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:53:26.083 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:53:26.082+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:53:30.052 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:30.053 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:30.053 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:30.058 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:53:30.466 INFO:tasks.workunit.client.0.smithi105.stderr:2023-10-14T00:53:30.466+0000 7f7a3c945640 0 --2- 172.21.15.105:0/3211689163 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a30020270 0x7f7a300af3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-14T00:53:31.317 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:53:31.317 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:53:31.318 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244811 2023-10-14T00:53:31.318 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:31.318 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:53:31.318 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:53:31.319 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:53:36.560 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.558+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:53:36.561 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.558+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:53:36.561 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.558+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:53:36.573 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:53:36.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:53:36.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:53:36.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:53:36.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:53:36.575 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:53:36.575 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:53:36.575 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:36.570+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:53:41.323 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:53:41.323 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:41.323 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:42.224 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:53:42.239 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:53:42.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244822 2023-10-14T00:53:42.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:42.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:53:42.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:53:42.240 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:53:49.995 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:49.994+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:53:49.995 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:49.994+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:53:49.999 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:49.998+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:53:50.009 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.006+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:53:50.009 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.006+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:53:50.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:53:50.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:53:50.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:53:50.015 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:53:50.015 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:53:50.015 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:50.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:53:52.243 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:53:52.246 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:53:52.252 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:53:52.257 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:53:53.181 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:53:53.181 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:53:53.182 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244833 2023-10-14T00:53:53.182 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:53:53.182 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:53:53.182 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:53:53.182 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:53:59.001 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:58.998+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:53:59.009 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:59.006+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:53:59.010 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:59.006+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:53:59.013 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:59.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:53:59.013 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:59.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:53:59.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:59.010+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:53:59.025 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:53:59.022+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:54:03.183 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:54:03.186 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:03.200 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:03.209 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:04.113 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:54:04.113 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:54:04.114 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244844 2023-10-14T00:54:04.114 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:04.114 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:54:04.115 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:54:04.115 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:54:14.116 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:54:14.118 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:14.118 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:14.141 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:14.941 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:54:14.941 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:54:14.942 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244854 2023-10-14T00:54:14.942 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:14.942 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:54:14.942 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:54:14.943 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:54:24.943 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:54:24.945 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:24.945 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:24.945 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:25.900 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:54:25.901 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:54:25.902 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244865 2023-10-14T00:54:25.902 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:25.902 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:54:25.902 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:54:25.902 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:54:30.060 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:30.057+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:54:30.060 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:30.057+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:54:30.061 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:30.057+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:54:35.907 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:35.909 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:35.910 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:54:35.913 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:36.170 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:54:36.165+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:54:36.904 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:54:36.904 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:54:36.905 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244876 2023-10-14T00:54:36.905 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:36.906 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:54:36.906 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:54:36.906 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:54:43.779 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:43.777+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:54:43.779 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:43.777+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:54:43.783 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:43.781+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:54:43.783 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:43.781+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:54:43.789 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:43.785+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:54:43.789 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:43.785+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:54:44.795 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:44.781+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:54:44.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:44.801+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:54:44.804 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:44.801+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:54:44.815 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:44.813+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:54:46.906 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:54:46.912 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:46.913 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:47.827 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:54:47.828 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:54:47.829 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244887 2023-10-14T00:54:47.830 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:47.830 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:54:47.830 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:54:47.830 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:54:52.082 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:54:52.073+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:54:57.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:54:57.835 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:54:57.837 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:54:57.858 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:54:58.755 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:54:58.756 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:54:58.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244898 2023-10-14T00:54:58.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:54:58.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:54:58.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:54:58.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:55:07.808 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:55:07.808 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:55:07.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:55:07.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:55:07.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:55:07.809 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:55:07.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:55:07.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:55:07.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:55:07.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:55:07.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:55:07.811 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:55:07.811 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:07.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:55:08.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:55:08.765 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:08.766 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:08.768 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:09.615 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:55:09.615 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:55:09.616 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244909 2023-10-14T00:55:09.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:09.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:55:09.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:55:09.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:55:19.621 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:19.621 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:19.625 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:55:19.626 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:20.528 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:55:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:55:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244920 2023-10-14T00:55:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:55:20.530 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:55:20.530 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:55:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:55:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:30.567 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:30.580 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:31.412 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:55:31.412 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:55:31.413 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244931 2023-10-14T00:55:31.413 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:31.413 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:55:31.414 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:55:31.414 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:55:32.719 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:32.716+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:55:32.723 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:32.724+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:55:32.723 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:32.724+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:55:32.734 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:32.736+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:55:41.423 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:41.423 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:41.438 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:41.446 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:55:42.276 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:55:42.277 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:55:42.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244942 2023-10-14T00:55:42.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:42.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:55:42.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:55:42.278 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:55:46.025 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:55:46.024+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:55:52.283 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:55:52.286 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:55:52.286 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:55:52.292 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:55:53.202 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:55:53.202 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:55:53.203 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244953 2023-10-14T00:55:53.203 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:55:53.203 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:55:53.203 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:55:53.204 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:55:59.668 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.667+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:55:59.671 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.667+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:55:59.673 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.671+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:55:59.673 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.671+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:55:59.678 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.675+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:55:59.678 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.675+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:55:59.681 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.679+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:55:59.681 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:55:59.679+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:56:03.205 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:56:03.206 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:03.216 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:03.219 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:04.035 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:56:04.035 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:56:04.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244964 2023-10-14T00:56:04.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:04.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:56:04.036 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:56:04.037 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:56:09.851 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:09.851+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:56:09.861 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:09.859+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:56:09.861 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:09.859+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:56:09.863 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:09.863+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:56:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:56:14.047 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:14.047 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:14.056 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:15.029 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:56:15.029 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:56:15.030 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244975 2023-10-14T00:56:15.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:15.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:56:15.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:56:15.031 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:56:25.034 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:25.036 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:25.036 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:56:25.053 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:25.982 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:56:25.982 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:56:25.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244985 2023-10-14T00:56:25.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:25.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:56:25.983 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:56:25.984 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:56:30.736 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:30.735+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:56:30.744 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:30.743+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:56:30.753 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:30.751+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:56:30.753 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:30.751+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:56:30.755 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:30.755+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:56:30.755 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:56:30.755+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:56:35.986 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:56:35.987 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:35.989 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:36.002 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:36.937 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:56:36.937 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:56:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697244996 2023-10-14T00:56:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:56:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:56:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:56:46.939 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:56:46.942 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:46.942 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:46.963 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:47.802 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:56:47.802 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:56:47.803 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245007 2023-10-14T00:56:47.803 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:47.803 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:56:47.803 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:56:47.804 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:56:52.560 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:56:52.559+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:56:52.560 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:56:52.559+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:56:54.705 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:56:54.703+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:56:57.805 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:56:57.806 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:56:57.807 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:56:57.811 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:56:58.647 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:56:58.654 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:56:58.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245018 2023-10-14T00:56:58.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:56:58.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:56:58.655 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:56:58.656 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:57:08.657 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:57:08.657 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:08.657 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:08.660 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:09.635 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:57:09.635 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:57:09.636 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245029 2023-10-14T00:57:09.636 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:09.636 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:57:09.636 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:57:09.637 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:57:19.638 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:57:19.638 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:19.638 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:19.639 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:20.475 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:57:20.476 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:57:20.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245040 2023-10-14T00:57:20.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:20.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:57:20.477 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:57:20.478 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:57:30.479 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:30.479 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:30.482 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:30.485 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:57:31.516 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:57:31.517 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:57:31.519 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245051 2023-10-14T00:57:31.519 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:31.519 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:57:31.519 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:57:31.520 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:57:41.525 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:41.529 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:41.546 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:57:41.549 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:42.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:57:42.618 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:57:42.618 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245062 2023-10-14T00:57:42.619 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:42.619 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:57:42.619 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:57:42.619 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:57:49.389 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:57:49.386+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:57:49.392 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:57:49.390+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:57:52.620 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:57:52.622 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:57:52.622 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:57:52.623 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:57:53.587 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:57:53.594 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:57:53.595 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245073 2023-10-14T00:57:53.595 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:57:53.595 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:57:53.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:57:53.596 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:58:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:58:03.612 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:03.615 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:03.633 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:04.518 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:58:04.522 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:58:04.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245084 2023-10-14T00:58:04.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:04.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:58:04.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:58:04.528 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:58:06.203 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:06.189+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:58:06.205 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:06.193+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:58:06.209 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:06.205+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:58:14.528 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:58:14.529 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:14.529 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:14.530 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:15.474 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:58:15.474 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:58:15.475 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245095 2023-10-14T00:58:15.475 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:15.475 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:58:15.475 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:58:15.476 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:58:25.476 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:58:25.478 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:25.488 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:26.434 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:58:26.435 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:58:26.436 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245106 2023-10-14T00:58:26.436 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:26.436 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:58:26.436 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:58:26.436 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:58:27.876 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:27.873+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:58:27.894 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:27.889+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:58:36.218 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:36.213+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:58:36.235 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:36.233+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:58:36.235 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:36.233+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:58:36.237 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:58:36.233+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:58:36.436 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:58:36.437 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:36.440 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:37.359 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:58:37.365 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:58:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245117 2023-10-14T00:58:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:58:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:58:37.368 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:58:47.373 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:47.382 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:58:47.383 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:47.389 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:48.318 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:58:48.326 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:58:48.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245128 2023-10-14T00:58:48.327 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:48.328 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:58:48.328 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:58:48.328 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:58:57.811 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:58:57.812+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:58:58.330 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:58:58.330 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:58:58.331 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:58:58.346 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:58:59.303 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:58:59.303 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:58:59.304 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245139 2023-10-14T00:58:59.304 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:58:59.304 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:58:59.304 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:58:59.305 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:59:03.155 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:59:03.156+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T00:59:03.155 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:59:03.156+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T00:59:03.155 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T00:59:03.156+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T00:59:09.308 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:59:09.309 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:09.315 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:09.331 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:59:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:59:10.162 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245150 2023-10-14T00:59:10.162 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:10.162 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:59:10.162 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:59:10.163 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:59:13.617 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.616+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:59:13.618 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.616+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:59:13.618 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.616+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:59:13.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.620+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:59:13.621 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.620+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:59:13.631 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.632+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:59:13.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.632+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:59:13.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.632+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:59:13.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.632+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:59:13.632 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.632+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:59:13.633 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:13.632+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:59:20.164 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:20.164 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:59:20.167 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:20.196 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:21.058 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:59:21.059 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:59:21.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245161 2023-10-14T00:59:21.059 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:21.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:59:21.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:59:21.060 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:59:31.062 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:31.068 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:31.070 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:31.071 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:59:31.977 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:59:31.978 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:59:31.979 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245171 2023-10-14T00:59:31.979 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:31.979 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:59:31.979 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:59:31.979 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:59:34.342 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T00:59:34.342 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T00:59:34.342 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T00:59:34.343 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T00:59:34.343 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T00:59:34.343 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T00:59:34.343 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T00:59:34.343 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T00:59:34.344 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T00:59:34.344 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T00:59:34.344 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T00:59:34.344 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T00:59:34.344 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T00:59:34.339+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T00:59:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:41.993 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:59:41.995 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:42.002 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:42.910 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:59:42.910 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:59:42.911 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245182 2023-10-14T00:59:42.911 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:42.911 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:59:42.912 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:59:42.912 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T00:59:52.919 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T00:59:52.926 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T00:59:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T00:59:52.942 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T00:59:53.883 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T00:59:53.884 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T00:59:53.890 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245193 2023-10-14T00:59:53.890 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T00:59:53.891 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T00:59:53.891 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T00:59:53.891 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:00:03.897 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:03.897 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:03.898 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:03.915 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:00:04.795 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:00:04.795 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:00:04.795 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245204 2023-10-14T01:00:04.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:04.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:00:04.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:00:04.796 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:00:14.799 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:14.799 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:14.800 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:00:14.802 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:15.777 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:00:15.777 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:00:15.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245215 2023-10-14T01:00:15.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:15.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:00:15.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:00:15.784 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:00:16.268 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:16.259+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:00:16.269 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:16.259+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:00:16.271 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:16.271+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:00:16.272 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:16.271+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:00:16.272 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:16.271+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:00:19.350 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:19.351+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:00:25.785 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:00:25.785 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:25.788 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:25.790 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:26.769 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:00:26.769 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:00:26.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245226 2023-10-14T01:00:26.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:26.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:00:26.770 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:00:26.771 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:00:28.759 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:00:28.758+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:00:28.759 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:00:28.758+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:00:29.472 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:29.470+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:00:29.472 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:29.470+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:00:29.479 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:29.478+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:00:34.790 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.786+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:00:34.790 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.786+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:00:34.802 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.794+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:00:34.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.802+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:00:34.811 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.802+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:00:34.813 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.810+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:00:34.814 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.810+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:00:34.814 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.810+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:00:34.814 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.810+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:00:34.814 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.810+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:00:34.814 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:34.810+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:00:35.977 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:35.974+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:00:35.982 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:35.982+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:00:35.986 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:00:35.986+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:00:36.771 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:36.772 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:36.789 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:36.805 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:00:37.700 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:00:37.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:00:37.700 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245237 2023-10-14T01:00:37.701 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:37.701 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:00:37.701 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:00:37.701 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:00:41.414 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:00:41.410+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:00:41.415 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:00:41.410+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:00:47.702 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:00:47.702 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:47.703 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:47.710 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:48.616 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:00:48.616 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:00:48.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245248 2023-10-14T01:00:48.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:48.617 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:00:48.618 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:00:48.618 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:00:58.624 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:00:58.630 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:00:58.634 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:00:58.649 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:00:59.548 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:00:59.548 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:00:59.548 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245259 2023-10-14T01:00:59.549 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:00:59.549 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:00:59.549 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:00:59.549 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:01:09.552 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:01:09.553 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:09.558 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:09.577 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:10.468 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:01:10.470 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:01:10.471 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245270 2023-10-14T01:01:10.471 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:10.472 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:01:10.472 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:01:10.472 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:01:17.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:01:17.877+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:01:17.884 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:01:17.877+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:01:20.475 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:20.477 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:20.478 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:20.490 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:01:21.451 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:01:21.451 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:01:21.451 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245281 2023-10-14T01:01:21.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:21.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:01:21.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:01:21.452 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:01:31.453 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:31.453 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:31.453 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:31.462 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:01:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:01:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:01:32.461 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245292 2023-10-14T01:01:32.461 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:32.461 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:01:32.462 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:01:32.462 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:01:42.467 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:42.473 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:42.474 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:01:42.486 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:43.393 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:01:43.394 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:01:43.395 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245303 2023-10-14T01:01:43.395 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:43.396 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:01:43.396 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:01:43.396 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:01:53.397 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:01:53.403 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:01:53.403 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:01:53.414 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:01:54.316 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:01:54.323 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:01:54.325 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245314 2023-10-14T01:01:54.326 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:01:54.326 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:01:54.326 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:01:54.326 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:02:04.336 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:02:04.337 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:04.346 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:04.353 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:05.366 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:02:05.367 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:02:05.367 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245325 2023-10-14T01:02:05.368 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:05.368 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:02:05.368 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:02:05.368 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:02:15.370 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:15.371 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:15.374 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:02:15.376 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:16.400 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:02:16.400 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:02:16.401 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245336 2023-10-14T01:02:16.401 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:02:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:02:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:02:20.801 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.800+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:02:20.810 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.808+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:02:20.818 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.816+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:02:20.819 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.816+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:02:20.823 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:02:20.823 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:02:20.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:02:20.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:02:20.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:02:20.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:02:20.824 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:02:20.825 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:02:20.825 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:20.824+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:02:26.415 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:02:26.416 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:26.416 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:26.426 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:27.228 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:02:27.229 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:02:27.229 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245347 2023-10-14T01:02:27.230 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:27.230 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:02:27.230 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:02:27.230 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:02:33.415 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:33.404+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:02:33.415 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:33.408+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:02:33.415 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:33.408+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:02:33.415 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:33.412+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:02:37.241 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:02:37.241 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:37.247 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:37.251 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:38.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:02:38.309 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:02:38.309 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245358 2023-10-14T01:02:38.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:38.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:02:38.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:02:38.310 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:02:41.562 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:41.552+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:02:41.562 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:02:41.560+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:02:48.312 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:48.313 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:02:48.313 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:02:48.313 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:49.177 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:02:49.177 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:02:49.178 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245369 2023-10-14T01:02:49.178 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:02:49.178 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:02:49.179 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:02:49.179 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:02:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:02:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:02:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:02:59.194 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:00.038 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:03:00.039 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:03:00.039 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245380 2023-10-14T01:03:00.040 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:00.040 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:03:00.040 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:03:00.040 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:03:10.047 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:10.050 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:10.055 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:03:10.073 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:10.985 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:03:10.985 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:03:10.986 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245390 2023-10-14T01:03:10.986 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:10.986 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:03:10.986 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:03:10.987 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:03:20.988 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:20.989 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:20.989 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:03:20.991 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:21.826 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:03:21.827 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:03:21.827 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245401 2023-10-14T01:03:21.828 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:21.828 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:03:21.828 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:03:21.828 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:03:31.829 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:03:31.829 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:31.833 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:31.839 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:33.009 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:03:33.050 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:03:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245413 2023-10-14T01:03:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:03:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:03:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:03:40.478 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:03:40.475+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:03:40.479 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:03:40.475+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:03:40.479 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:03:40.475+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:03:40.479 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:03:40.475+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:03:40.480 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:03:40.475+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:03:40.480 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:03:40.475+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:03:43.034 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:03:43.038 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:43.040 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:43.048 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:43.821 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:03:43.822 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:03:43.823 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245423 2023-10-14T01:03:43.823 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:43.823 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:03:43.824 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:03:43.824 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:03:53.830 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:03:53.831 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:03:53.832 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:03:53.833 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:03:54.755 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:03:54.755 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:03:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245434 2023-10-14T01:03:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:03:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:03:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:03:54.757 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:04:01.099 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:04:01.091+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:04:01.099 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:04:01.091+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:04:04.764 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:04:04.766 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:04.768 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:04.778 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:04:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:04:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245445 2023-10-14T01:04:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:04:05.699 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:04:05.699 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:04:15.703 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:04:15.703 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:15.709 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:15.711 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:15.886 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:04:15.887 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:04:15.887 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:04:15.887 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:04:15.887 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:04:15.887 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:04:15.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:04:15.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:04:15.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:04:15.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:04:15.888 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:04:15.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:04:15.889 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:04:15.882+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:04:16.701 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:04:16.701 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:04:16.702 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245456 2023-10-14T01:04:16.702 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:16.702 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:04:16.703 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:04:16.703 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:04:26.703 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:04:26.704 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:26.705 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:26.706 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:27.681 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:04:27.681 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:04:27.682 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245467 2023-10-14T01:04:27.682 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:27.682 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:04:27.682 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:04:27.683 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:04:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:04:37.700 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:37.714 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:37.716 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:38.574 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:04:38.575 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:04:38.576 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245478 2023-10-14T01:04:38.576 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:38.576 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:04:38.576 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:04:38.577 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:04:47.760 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:04:47.758+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:04:47.760 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:04:47.758+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:04:48.582 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:04:48.583 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:48.589 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:04:48.599 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:49.586 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:04:49.586 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:04:49.587 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245489 2023-10-14T01:04:49.587 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:04:49.587 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:04:49.587 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:04:49.587 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:04:59.588 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:04:59.590 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:04:59.602 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:04:59.618 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:00.381 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:05:00.381 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:05:00.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245500 2023-10-14T01:05:00.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:00.382 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:05:00.383 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:05:00.383 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:05:10.385 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:05:10.385 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:10.386 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:10.392 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:11.272 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:05:11.272 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:05:11.273 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245511 2023-10-14T01:05:11.273 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:11.273 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:05:11.273 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:05:11.274 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:05:21.275 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:21.275 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:21.278 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:05:21.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:22.270 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:05:22.274 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:05:22.275 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245522 2023-10-14T01:05:22.276 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:22.276 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:05:22.276 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:05:22.276 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:05:32.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:32.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:32.280 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:05:32.296 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:33.195 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:05:33.198 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:05:33.199 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245533 2023-10-14T01:05:33.199 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:33.199 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:05:33.200 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:05:33.200 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:05:43.202 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:05:43.203 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:43.205 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:43.207 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:44.117 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:05:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:05:44.119 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245544 2023-10-14T01:05:44.119 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:44.119 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:05:44.119 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:05:44.119 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:05:54.120 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:05:54.121 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:05:54.137 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:05:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:05:55.128 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:05:55.128 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:05:55.129 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245555 2023-10-14T01:05:55.129 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:05:55.129 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ -n 172.21.15.105:0/1796457399 ]] 2023-10-14T01:05:55.130 INFO:tasks.workunit.client.0.smithi105.stderr:+ [[ 172.21.15.105:0/1796457399 != \1\7\2\.\2\1\.\1\5\.\1\0\5\:\0\/\1\7\9\6\4\5\7\3\9\9 ]] 2023-10-14T01:05:55.130 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:06:05.141 INFO:tasks.workunit.client.0.smithi105.stderr:++ ceph mgr dump 2023-10-14T01:06:05.142 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-14T01:06:05.142 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq '.active_clients[]' 2023-10-14T01:06:05.147 INFO:tasks.workunit.client.0.smithi105.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-14T01:06:06.001 INFO:tasks.workunit.client.0.smithi105.stderr:+ CLIENT_ADDR=172.21.15.105:0/1796457399 2023-10-14T01:06:06.001 INFO:tasks.workunit.client.0.smithi105.stderr:++ date +%s 2023-10-14T01:06:06.002 INFO:tasks.workunit.client.0.smithi105.stderr:+ CURRENT_TIME=1697245566 2023-10-14T01:06:06.002 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( CURRENT_TIME <= END_TIME )) 2023-10-14T01:06:06.003 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i = 1 )) 2023-10-14T01:06:06.003 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:06:06.003 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:06:06.181 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:06:06.180+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:06:06.182 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:06:06.198 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:06:13.572 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.572+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:06:13.572 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.572+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:06:13.572 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.572+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:06:13.572 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.572+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:06:13.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:06:13.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:06:13.574 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:06:13.575 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:06:13.576 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:06:13.576 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:06:13.576 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:06:13.576 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:06:13.576 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:06:13.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:06:16.200 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:06:16.200 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:06:16.201 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:06:16.291 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:06:16.292+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:06:16.291 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:06:16.302 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:06:26.304 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:06:26.305 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:06:26.305 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:06:26.394 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:06:26.396+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:06:26.398 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:06:26.407 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:06:36.408 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:06:36.409 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:06:36.409 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:06:36.490 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:06:36.492+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:06:36.491 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:06:36.514 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:06:46.516 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:06:46.516 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:06:46.517 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:06:46.649 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:06:46.648+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:06:46.651 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:06:46.669 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:06:56.671 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:06:56.671 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:06:56.672 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:06:56.783 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:06:56.783+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:06:56.784 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:06:56.795 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:07:06.798 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:07:06.798 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:07:06.799 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:06.915 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:07:06.915+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:06.916 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:06.950 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:07:10.113 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:07:10.107+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:07:16.951 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:07:16.952 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:07:16.952 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:17.099 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:07:17.099+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:17.100 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:17.107 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:07:26.554 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.551+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:07:26.566 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:07:26.566 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:07:26.566 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:07:26.566 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:07:26.566 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:07:26.567 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:07:26.567 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:07:26.567 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:07:26.567 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:07:26.567 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:07:26.567 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:07:26.568 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:07:26.563+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:07:27.109 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:07:27.110 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:07:27.110 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:27.206 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:07:27.207+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:27.211 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:27.223 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:07:37.224 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:07:37.225 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:07:37.225 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:37.334 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:07:37.335+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:37.335 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:37.344 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:07:47.346 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:07:47.346 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:07:47.346 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:47.421 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:07:47.419+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:47.422 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:47.435 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:07:57.436 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:07:57.437 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:07:57.437 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:07:57.514 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:07:57.510+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:07:57.514 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:07:57.525 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:08:07.526 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:08:07.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:08:07.527 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:07.622 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:08:07.622+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:07.623 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:07.636 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:08:17.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:08:17.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:08:17.638 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:17.729 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:08:17.726+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:17.729 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:08:27.734 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:08:27.734 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:08:27.735 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:27.810 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:08:27.810+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:27.811 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:27.815 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:08:37.819 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:08:37.819 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:08:37.819 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:37.940 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:08:37.938+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:37.941 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:37.967 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:08:46.346 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:08:46.346+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:08:46.346 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:08:46.346+0000 7f120e762640 -1 reset not still connected to 0x5632b470f2b0 2023-10-14T01:08:46.347 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:08:46.346+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:08:47.969 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:08:47.969 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:08:47.969 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:48.090 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:08:48.090+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:48.091 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:48.095 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:08:58.099 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:08:58.099 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:08:58.099 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:08:58.208 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:08:58.205+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:08:58.209 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:08:58.222 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:09:08.224 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:09:08.225 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:09:08.225 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:08.300 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:09:08.297+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:08.300 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:08.307 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:09:18.311 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:09:18.312 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:09:18.312 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:18.401 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:09:18.397+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:18.401 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:18.408 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:09:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:09:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:09:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:28.546 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:09:28.545+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:28.547 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:28.560 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:09:37.719 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:09:37.719 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:09:37.720 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:09:37.720 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:09:37.720 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:09:37.720 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:09:37.720 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:09:37.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:09:37.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:09:37.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:09:37.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:09:37.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:09:37.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:09:37.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:09:38.561 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:09:38.562 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:09:38.562 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:38.863 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:09:38.861+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:38.864 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:38.878 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:09:48.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:09:48.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:09:48.880 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:48.983 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:09:48.985+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:48.984 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:48.989 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:09:58.990 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:09:58.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:09:58.991 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 2m 2023-10-14T01:09:59.063 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:09:59.064+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:09:59.064 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule add failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:09:59.069 INFO:tasks.workunit.client.0.smithi105.stderr:+ sleep 10 2023-10-14T01:10:09.072 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i++ )) 2023-10-14T01:10:09.072 INFO:tasks.workunit.client.0.smithi105.stderr:+ (( i <= 24 )) 2023-10-14T01:10:09.072 INFO:tasks.workunit.client.0.smithi105.stderr:+ grep 'every 2m' 2023-10-14T01:10:09.076 INFO:tasks.workunit.client.0.smithi105.stderr:+ rbd mirror snapshot schedule ls -p rbd --image image1 2023-10-14T01:10:09.151 INFO:tasks.workunit.client.0.smithi105.stderr:rbd: rbd mirror snapshot schedule list failed: (11) Resource temporarily unavailable: rbd_support module is not ready, try again 2023-10-14T01:10:09.152 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-10-14T01:10:09.152+0000 7f93c0f9e640 -1 mgr.server reply reply (11) Resource temporarily unavailable rbd_support module is not ready, try again 2023-10-14T01:10:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T01:10:09.169 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-14T01:10:09.169 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-14T01:10:09.917 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-14T01:10:09.980 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f97ca0e4b1f64dda8a845dff817b51f0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_idryomov_ceph_735f68d7bf91d643813a8420fc2b11441f4bd8c4/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-14T01:10:09.982 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T01:10:10.030 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-14T01:10:10.042 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T01:10:10.043 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:10.527 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:10.528 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:10:10.558 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2048,"stamp":"2023-10-14T01:10:10.342155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28350562124,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1839208,"num_read_kb":7165870,"num_write":1813973,"num_write_kb":7254903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85535,"ondisk_log_size":85535,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19581700,"kb_used_data":14506792,"kb_used_omap":1990,"kb_used_meta":5072889,"kb_avail":263533820,"statfs":{"total":289910292480,"available":269858631680,"internally_reserved":0,"allocated":14854955008,"data_stored":17921154863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5194639175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,5,8,9,18,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18554880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4647,"num_read_kb":18520,"num_write":4542,"num_write_kb":18168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002081"},"pg_stats":[{"pgid":"2.7","version":"85'9412","reported_seq":13069,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.743040+0000","last_change":"2023-10-14T00:04:47.892979+0000","last_active":"2023-10-14T01:10:07.743040+0000","last_peered":"2023-10-14T01:10:07.743040+0000","last_clean":"2023-10-14T01:10:07.743040+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:10:07.743040+0000","last_undegraded":"2023-10-14T01:10:07.743040+0000","last_fullsized":"2023-10-14T01:10:07.743040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:19:15.695751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6333","reported_seq":9718,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.729247+0000","last_change":"2023-10-14T00:04:47.892672+0000","last_active":"2023-10-14T01:10:07.729247+0000","last_peered":"2023-10-14T01:10:07.729247+0000","last_clean":"2023-10-14T01:10:07.729247+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:10:07.729247+0000","last_undegraded":"2023-10-14T01:10:07.729247+0000","last_fullsized":"2023-10-14T01:10:07.729247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:59:03.121676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'12","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:09:17.040477+0000","last_change":"2023-10-14T00:04:47.893288+0000","last_active":"2023-10-14T01:09:17.040477+0000","last_peered":"2023-10-14T01:09:17.040477+0000","last_clean":"2023-10-14T01:09:17.040477+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:09:17.040477+0000","last_undegraded":"2023-10-14T01:09:17.040477+0000","last_fullsized":"2023-10-14T01:09:17.040477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:05:35.203825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'12507","reported_seq":17754,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.751545+0000","last_change":"2023-10-14T00:04:47.892782+0000","last_active":"2023-10-14T01:10:07.751545+0000","last_peered":"2023-10-14T01:10:07.751545+0000","last_clean":"2023-10-14T01:10:07.751545+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:10:07.751545+0000","last_undegraded":"2023-10-14T01:10:07.751545+0000","last_fullsized":"2023-10-14T01:10:07.751545+0000","mapping_epoch":12,"log_start":"65'2500","ondisk_log_start":"65'2500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:57.647677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'449579","reported_seq":894940,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:06.786971+0000","last_change":"2023-10-14T00:04:55.935661+0000","last_active":"2023-10-14T01:10:06.786971+0000","last_peered":"2023-10-14T01:10:06.786971+0000","last_clean":"2023-10-14T01:10:06.786971+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:10:06.786971+0000","last_undegraded":"2023-10-14T01:10:06.786971+0000","last_fullsized":"2023-10-14T01:10:06.786971+0000","mapping_epoch":15,"log_start":"84'439529","ondisk_log_start":"84'439529","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:11:39.347355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":6983196672,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":447746,"num_read_kb":1761332,"num_write":447057,"num_write_kb":1788228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'9259","reported_seq":12801,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.329603+0000","last_change":"2023-10-14T00:04:47.890020+0000","last_active":"2023-10-14T01:10:05.329603+0000","last_peered":"2023-10-14T01:10:05.329603+0000","last_clean":"2023-10-14T01:10:05.329603+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:10:05.329603+0000","last_undegraded":"2023-10-14T01:10:05.329603+0000","last_fullsized":"2023-10-14T01:10:05.329603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:42:10.640763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'457025","reported_seq":909019,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.846122+0000","last_change":"2023-10-14T00:04:55.940744+0000","last_active":"2023-10-14T01:10:07.846122+0000","last_peered":"2023-10-14T01:10:07.846122+0000","last_clean":"2023-10-14T01:10:07.846122+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:10:07.846122+0000","last_undegraded":"2023-10-14T01:10:07.846122+0000","last_fullsized":"2023-10-14T01:10:07.846122+0000","mapping_epoch":15,"log_start":"84'446925","ondisk_log_start":"84'446925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:03:23.735983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7113400320,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":454423,"num_read_kb":1787264,"num_write":454455,"num_write_kb":1817820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'139","reported_seq":3646,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.330160+0000","last_change":"2023-10-14T00:04:47.889952+0000","last_active":"2023-10-14T01:10:05.330160+0000","last_peered":"2023-10-14T01:10:05.330160+0000","last_clean":"2023-10-14T01:10:05.330160+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:10:05.330160+0000","last_undegraded":"2023-10-14T01:10:05.330160+0000","last_fullsized":"2023-10-14T01:10:05.330160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:33:09.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'459653","reported_seq":915570,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:06.806933+0000","last_change":"2023-10-14T00:04:55.935824+0000","last_active":"2023-10-14T01:10:06.806933+0000","last_peered":"2023-10-14T01:10:06.806933+0000","last_clean":"2023-10-14T01:10:06.806933+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:10:06.806933+0000","last_undegraded":"2023-10-14T01:10:06.806933+0000","last_fullsized":"2023-10-14T01:10:06.806933+0000","mapping_epoch":15,"log_start":"84'449624","ondisk_log_start":"84'449624","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:29:57.244648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7135670272,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":458352,"num_read_kb":1803592,"num_write":457077,"num_write_kb":1828308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'55","reported_seq":1308,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:00.210168+0000","last_change":"2023-10-14T00:04:47.890489+0000","last_active":"2023-10-14T01:10:00.210168+0000","last_peered":"2023-10-14T01:10:00.210168+0000","last_clean":"2023-10-14T01:10:00.210168+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:10:00.210168+0000","last_undegraded":"2023-10-14T01:10:00.210168+0000","last_fullsized":"2023-10-14T01:10:00.210168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:36:13.665649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'457146","reported_seq":910964,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.855510+0000","last_change":"2023-10-14T00:04:55.942378+0000","last_active":"2023-10-14T01:10:07.855510+0000","last_peered":"2023-10-14T01:10:07.855510+0000","last_clean":"2023-10-14T01:10:07.855510+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:10:07.855510+0000","last_undegraded":"2023-10-14T01:10:07.855510+0000","last_fullsized":"2023-10-14T01:10:07.855510+0000","mapping_epoch":15,"log_start":"84'447133","ondisk_log_start":"84'447133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:31:35.598409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7117701139,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":456250,"num_read_kb":1794768,"num_write":454574,"num_write_kb":1818290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'12429","reported_seq":17028,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.780555+0000","last_change":"2023-10-14T00:04:47.888394+0000","last_active":"2023-10-14T01:10:07.780555+0000","last_peered":"2023-10-14T01:10:07.780555+0000","last_clean":"2023-10-14T01:10:07.780555+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:10:07.780555+0000","last_undegraded":"2023-10-14T01:10:07.780555+0000","last_fullsized":"2023-10-14T01:10:07.780555+0000","mapping_epoch":12,"log_start":"65'2400","ondisk_log_start":"65'2400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:15:14.957936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":244,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-14T01:09:07.498152+0000","last_change":"2023-10-14T00:04:42.867239+0000","last_active":"2023-10-14T01:09:07.498152+0000","last_peered":"2023-10-14T01:09:07.498152+0000","last_clean":"2023-10-14T01:09:07.498152+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:09:07.498152+0000","last_undegraded":"2023-10-14T01:09:07.498152+0000","last_fullsized":"2023-10-14T01:09:07.498152+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_clean_scrub_stamp":"2023-10-14T00:04:41.840847+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:45:11.022601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28349968403,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1816771,"num_read_kb":7146956,"num_write":1813163,"num_write_kb":7252646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14853423104,"data_stored":17921450022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40192,"ondisk_log_size":40192,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45246,"ondisk_log_size":45246,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8684532,"kb_used_data":7251624,"kb_used_omap":976,"kb_used_meta":1431919,"kb_avail":85687308,"statfs":{"total":96636764160,"available":87743803392,"internally_reserved":0,"allocated":7425662976,"data_stored":8959192746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1466285341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.33900000000000002,"15min":0.40100000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.747,"5min":0.99199999999999999,"15min":4.6189999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.34699999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":3.4950000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.23300000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.42299999999999999,"15min":0.49399999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":1.2330000000000001,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.46400000000000002,"15min":0.51100000000000001},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":1.2629999999999999,"5min":4.2060000000000004,"15min":16.747},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739158,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5454340,"kb_used_data":3637628,"kb_used_omap":510,"kb_used_meta":1816193,"kb_avail":88917500,"statfs":{"total":96636764160,"available":91051520000,"internally_reserved":0,"allocated":3724931072,"data_stored":4492040745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1859781716},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,6,2,8,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":3.0209999999999999,"5min":4.117,"15min":5.2370000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.56499999999999995,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":0.46800000000000003,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":4.4740000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.623,"5min":5.6040000000000001,"15min":5.6040000000000001},"last":0.252},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42299999999999999,"15min":0.47299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.65300000000000002,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739160,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5442828,"kb_used_data":3617540,"kb_used_omap":502,"kb_used_meta":1824777,"kb_avail":88929012,"statfs":{"total":96636764160,"available":91063308288,"internally_reserved":0,"allocated":3704360960,"data_stored":4469921372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1868572118},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,0,4,6,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.40000000000000002,"15min":0.41799999999999998},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":4.452,"5min":4.452,"15min":8.3239999999999998},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.33800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.66700000000000004,"15min":0.56499999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.188},"max":{"1min":4.5449999999999999,"5min":5.875,"15min":26.074999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":4.492,"5min":4.492,"15min":6.5449999999999999},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3703341056,"data_stored":4469526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3723382784,"data_stored":4491210771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7426699264,"data_stored":8960712723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:10:10.560 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:11.093 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:11.094 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:10:11.124 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2048,"stamp":"2023-10-14T01:10:10.342155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28350562124,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1839208,"num_read_kb":7165870,"num_write":1813973,"num_write_kb":7254903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85535,"ondisk_log_size":85535,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19581700,"kb_used_data":14506792,"kb_used_omap":1990,"kb_used_meta":5072889,"kb_avail":263533820,"statfs":{"total":289910292480,"available":269858631680,"internally_reserved":0,"allocated":14854955008,"data_stored":17921154863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5194639175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,5,8,9,18,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18554880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4647,"num_read_kb":18520,"num_write":4542,"num_write_kb":18168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002081"},"pg_stats":[{"pgid":"2.7","version":"85'9412","reported_seq":13069,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.743040+0000","last_change":"2023-10-14T00:04:47.892979+0000","last_active":"2023-10-14T01:10:07.743040+0000","last_peered":"2023-10-14T01:10:07.743040+0000","last_clean":"2023-10-14T01:10:07.743040+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:10:07.743040+0000","last_undegraded":"2023-10-14T01:10:07.743040+0000","last_fullsized":"2023-10-14T01:10:07.743040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:19:15.695751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6333","reported_seq":9718,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.729247+0000","last_change":"2023-10-14T00:04:47.892672+0000","last_active":"2023-10-14T01:10:07.729247+0000","last_peered":"2023-10-14T01:10:07.729247+0000","last_clean":"2023-10-14T01:10:07.729247+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:10:07.729247+0000","last_undegraded":"2023-10-14T01:10:07.729247+0000","last_fullsized":"2023-10-14T01:10:07.729247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:59:03.121676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'12","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:09:17.040477+0000","last_change":"2023-10-14T00:04:47.893288+0000","last_active":"2023-10-14T01:09:17.040477+0000","last_peered":"2023-10-14T01:09:17.040477+0000","last_clean":"2023-10-14T01:09:17.040477+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:09:17.040477+0000","last_undegraded":"2023-10-14T01:09:17.040477+0000","last_fullsized":"2023-10-14T01:09:17.040477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:05:35.203825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'12507","reported_seq":17754,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.751545+0000","last_change":"2023-10-14T00:04:47.892782+0000","last_active":"2023-10-14T01:10:07.751545+0000","last_peered":"2023-10-14T01:10:07.751545+0000","last_clean":"2023-10-14T01:10:07.751545+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:10:07.751545+0000","last_undegraded":"2023-10-14T01:10:07.751545+0000","last_fullsized":"2023-10-14T01:10:07.751545+0000","mapping_epoch":12,"log_start":"65'2500","ondisk_log_start":"65'2500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:57.647677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'449579","reported_seq":894940,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:06.786971+0000","last_change":"2023-10-14T00:04:55.935661+0000","last_active":"2023-10-14T01:10:06.786971+0000","last_peered":"2023-10-14T01:10:06.786971+0000","last_clean":"2023-10-14T01:10:06.786971+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:10:06.786971+0000","last_undegraded":"2023-10-14T01:10:06.786971+0000","last_fullsized":"2023-10-14T01:10:06.786971+0000","mapping_epoch":15,"log_start":"84'439529","ondisk_log_start":"84'439529","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:11:39.347355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":6983196672,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":447746,"num_read_kb":1761332,"num_write":447057,"num_write_kb":1788228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'9259","reported_seq":12801,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.329603+0000","last_change":"2023-10-14T00:04:47.890020+0000","last_active":"2023-10-14T01:10:05.329603+0000","last_peered":"2023-10-14T01:10:05.329603+0000","last_clean":"2023-10-14T01:10:05.329603+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:10:05.329603+0000","last_undegraded":"2023-10-14T01:10:05.329603+0000","last_fullsized":"2023-10-14T01:10:05.329603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:42:10.640763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'457025","reported_seq":909019,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.846122+0000","last_change":"2023-10-14T00:04:55.940744+0000","last_active":"2023-10-14T01:10:07.846122+0000","last_peered":"2023-10-14T01:10:07.846122+0000","last_clean":"2023-10-14T01:10:07.846122+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:10:07.846122+0000","last_undegraded":"2023-10-14T01:10:07.846122+0000","last_fullsized":"2023-10-14T01:10:07.846122+0000","mapping_epoch":15,"log_start":"84'446925","ondisk_log_start":"84'446925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:03:23.735983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7113400320,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":454423,"num_read_kb":1787264,"num_write":454455,"num_write_kb":1817820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'139","reported_seq":3646,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.330160+0000","last_change":"2023-10-14T00:04:47.889952+0000","last_active":"2023-10-14T01:10:05.330160+0000","last_peered":"2023-10-14T01:10:05.330160+0000","last_clean":"2023-10-14T01:10:05.330160+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:10:05.330160+0000","last_undegraded":"2023-10-14T01:10:05.330160+0000","last_fullsized":"2023-10-14T01:10:05.330160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:33:09.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'459653","reported_seq":915570,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:06.806933+0000","last_change":"2023-10-14T00:04:55.935824+0000","last_active":"2023-10-14T01:10:06.806933+0000","last_peered":"2023-10-14T01:10:06.806933+0000","last_clean":"2023-10-14T01:10:06.806933+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:10:06.806933+0000","last_undegraded":"2023-10-14T01:10:06.806933+0000","last_fullsized":"2023-10-14T01:10:06.806933+0000","mapping_epoch":15,"log_start":"84'449624","ondisk_log_start":"84'449624","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:29:57.244648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7135670272,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":458352,"num_read_kb":1803592,"num_write":457077,"num_write_kb":1828308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'55","reported_seq":1308,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:00.210168+0000","last_change":"2023-10-14T00:04:47.890489+0000","last_active":"2023-10-14T01:10:00.210168+0000","last_peered":"2023-10-14T01:10:00.210168+0000","last_clean":"2023-10-14T01:10:00.210168+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:10:00.210168+0000","last_undegraded":"2023-10-14T01:10:00.210168+0000","last_fullsized":"2023-10-14T01:10:00.210168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:36:13.665649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'457146","reported_seq":910964,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.855510+0000","last_change":"2023-10-14T00:04:55.942378+0000","last_active":"2023-10-14T01:10:07.855510+0000","last_peered":"2023-10-14T01:10:07.855510+0000","last_clean":"2023-10-14T01:10:07.855510+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:10:07.855510+0000","last_undegraded":"2023-10-14T01:10:07.855510+0000","last_fullsized":"2023-10-14T01:10:07.855510+0000","mapping_epoch":15,"log_start":"84'447133","ondisk_log_start":"84'447133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:31:35.598409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7117701139,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":456250,"num_read_kb":1794768,"num_write":454574,"num_write_kb":1818290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'12429","reported_seq":17028,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.780555+0000","last_change":"2023-10-14T00:04:47.888394+0000","last_active":"2023-10-14T01:10:07.780555+0000","last_peered":"2023-10-14T01:10:07.780555+0000","last_clean":"2023-10-14T01:10:07.780555+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:10:07.780555+0000","last_undegraded":"2023-10-14T01:10:07.780555+0000","last_fullsized":"2023-10-14T01:10:07.780555+0000","mapping_epoch":12,"log_start":"65'2400","ondisk_log_start":"65'2400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:15:14.957936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":244,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-14T01:09:07.498152+0000","last_change":"2023-10-14T00:04:42.867239+0000","last_active":"2023-10-14T01:09:07.498152+0000","last_peered":"2023-10-14T01:09:07.498152+0000","last_clean":"2023-10-14T01:09:07.498152+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:09:07.498152+0000","last_undegraded":"2023-10-14T01:09:07.498152+0000","last_fullsized":"2023-10-14T01:09:07.498152+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_clean_scrub_stamp":"2023-10-14T00:04:41.840847+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:45:11.022601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28349968403,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1816771,"num_read_kb":7146956,"num_write":1813163,"num_write_kb":7252646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14853423104,"data_stored":17921450022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40192,"ondisk_log_size":40192,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45246,"ondisk_log_size":45246,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8684532,"kb_used_data":7251624,"kb_used_omap":976,"kb_used_meta":1431919,"kb_avail":85687308,"statfs":{"total":96636764160,"available":87743803392,"internally_reserved":0,"allocated":7425662976,"data_stored":8959192746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1466285341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.33900000000000002,"15min":0.40100000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.747,"5min":0.99199999999999999,"15min":4.6189999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.34699999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":3.4950000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.23300000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.42299999999999999,"15min":0.49399999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":1.2330000000000001,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.46400000000000002,"15min":0.51100000000000001},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":1.2629999999999999,"5min":4.2060000000000004,"15min":16.747},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739158,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5454340,"kb_used_data":3637628,"kb_used_omap":510,"kb_used_meta":1816193,"kb_avail":88917500,"statfs":{"total":96636764160,"available":91051520000,"internally_reserved":0,"allocated":3724931072,"data_stored":4492040745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1859781716},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,6,2,8,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":3.0209999999999999,"5min":4.117,"15min":5.2370000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.56499999999999995,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":0.46800000000000003,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":4.4740000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.623,"5min":5.6040000000000001,"15min":5.6040000000000001},"last":0.252},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42299999999999999,"15min":0.47299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.65300000000000002,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739160,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5442828,"kb_used_data":3617540,"kb_used_omap":502,"kb_used_meta":1824777,"kb_avail":88929012,"statfs":{"total":96636764160,"available":91063308288,"internally_reserved":0,"allocated":3704360960,"data_stored":4469921372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1868572118},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,0,4,6,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.40000000000000002,"15min":0.41799999999999998},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":4.452,"5min":4.452,"15min":8.3239999999999998},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.33800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.66700000000000004,"15min":0.56499999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.188},"max":{"1min":4.5449999999999999,"5min":5.875,"15min":26.074999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":4.492,"5min":4.492,"15min":6.5449999999999999},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3703341056,"data_stored":4469526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3723382784,"data_stored":4491210771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7426699264,"data_stored":8960712723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:10:11.125 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T01:10:11.125 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:11.600 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:11.601 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:10:11.639 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2048,"stamp":"2023-10-14T01:10:10.342155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28350562124,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1839208,"num_read_kb":7165870,"num_write":1813973,"num_write_kb":7254903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85535,"ondisk_log_size":85535,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19581700,"kb_used_data":14506792,"kb_used_omap":1990,"kb_used_meta":5072889,"kb_avail":263533820,"statfs":{"total":289910292480,"available":269858631680,"internally_reserved":0,"allocated":14854955008,"data_stored":17921154863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5194639175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,5,8,9,18,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18554880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4647,"num_read_kb":18520,"num_write":4542,"num_write_kb":18168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002081"},"pg_stats":[{"pgid":"2.7","version":"85'9412","reported_seq":13069,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.743040+0000","last_change":"2023-10-14T00:04:47.892979+0000","last_active":"2023-10-14T01:10:07.743040+0000","last_peered":"2023-10-14T01:10:07.743040+0000","last_clean":"2023-10-14T01:10:07.743040+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:10:07.743040+0000","last_undegraded":"2023-10-14T01:10:07.743040+0000","last_fullsized":"2023-10-14T01:10:07.743040+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:19:15.695751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6333","reported_seq":9718,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.729247+0000","last_change":"2023-10-14T00:04:47.892672+0000","last_active":"2023-10-14T01:10:07.729247+0000","last_peered":"2023-10-14T01:10:07.729247+0000","last_clean":"2023-10-14T01:10:07.729247+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:10:07.729247+0000","last_undegraded":"2023-10-14T01:10:07.729247+0000","last_fullsized":"2023-10-14T01:10:07.729247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:59:03.121676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'12","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:09:17.040477+0000","last_change":"2023-10-14T00:04:47.893288+0000","last_active":"2023-10-14T01:09:17.040477+0000","last_peered":"2023-10-14T01:09:17.040477+0000","last_clean":"2023-10-14T01:09:17.040477+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:09:17.040477+0000","last_undegraded":"2023-10-14T01:09:17.040477+0000","last_fullsized":"2023-10-14T01:09:17.040477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:05:35.203825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'12507","reported_seq":17754,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.751545+0000","last_change":"2023-10-14T00:04:47.892782+0000","last_active":"2023-10-14T01:10:07.751545+0000","last_peered":"2023-10-14T01:10:07.751545+0000","last_clean":"2023-10-14T01:10:07.751545+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:10:07.751545+0000","last_undegraded":"2023-10-14T01:10:07.751545+0000","last_fullsized":"2023-10-14T01:10:07.751545+0000","mapping_epoch":12,"log_start":"65'2500","ondisk_log_start":"65'2500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:57.647677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'449579","reported_seq":894940,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:06.786971+0000","last_change":"2023-10-14T00:04:55.935661+0000","last_active":"2023-10-14T01:10:06.786971+0000","last_peered":"2023-10-14T01:10:06.786971+0000","last_clean":"2023-10-14T01:10:06.786971+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:10:06.786971+0000","last_undegraded":"2023-10-14T01:10:06.786971+0000","last_fullsized":"2023-10-14T01:10:06.786971+0000","mapping_epoch":15,"log_start":"84'439529","ondisk_log_start":"84'439529","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:11:39.347355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":6983196672,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":447746,"num_read_kb":1761332,"num_write":447057,"num_write_kb":1788228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'9259","reported_seq":12801,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.329603+0000","last_change":"2023-10-14T00:04:47.890020+0000","last_active":"2023-10-14T01:10:05.329603+0000","last_peered":"2023-10-14T01:10:05.329603+0000","last_clean":"2023-10-14T01:10:05.329603+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:10:05.329603+0000","last_undegraded":"2023-10-14T01:10:05.329603+0000","last_fullsized":"2023-10-14T01:10:05.329603+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9259,"log_dups_size":0,"ondisk_log_size":9259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:42:10.640763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'457025","reported_seq":909019,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.846122+0000","last_change":"2023-10-14T00:04:55.940744+0000","last_active":"2023-10-14T01:10:07.846122+0000","last_peered":"2023-10-14T01:10:07.846122+0000","last_clean":"2023-10-14T01:10:07.846122+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:10:07.846122+0000","last_undegraded":"2023-10-14T01:10:07.846122+0000","last_fullsized":"2023-10-14T01:10:07.846122+0000","mapping_epoch":15,"log_start":"84'446925","ondisk_log_start":"84'446925","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:03:23.735983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7113400320,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":454423,"num_read_kb":1787264,"num_write":454455,"num_write_kb":1817820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'139","reported_seq":3646,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.330160+0000","last_change":"2023-10-14T00:04:47.889952+0000","last_active":"2023-10-14T01:10:05.330160+0000","last_peered":"2023-10-14T01:10:05.330160+0000","last_clean":"2023-10-14T01:10:05.330160+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:10:05.330160+0000","last_undegraded":"2023-10-14T01:10:05.330160+0000","last_fullsized":"2023-10-14T01:10:05.330160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:33:09.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'459653","reported_seq":915570,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:06.806933+0000","last_change":"2023-10-14T00:04:55.935824+0000","last_active":"2023-10-14T01:10:06.806933+0000","last_peered":"2023-10-14T01:10:06.806933+0000","last_clean":"2023-10-14T01:10:06.806933+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:10:06.806933+0000","last_undegraded":"2023-10-14T01:10:06.806933+0000","last_fullsized":"2023-10-14T01:10:06.806933+0000","mapping_epoch":15,"log_start":"84'449624","ondisk_log_start":"84'449624","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:29:57.244648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7135670272,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":458352,"num_read_kb":1803592,"num_write":457077,"num_write_kb":1828308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'55","reported_seq":1308,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:00.210168+0000","last_change":"2023-10-14T00:04:47.890489+0000","last_active":"2023-10-14T01:10:00.210168+0000","last_peered":"2023-10-14T01:10:00.210168+0000","last_clean":"2023-10-14T01:10:00.210168+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:10:00.210168+0000","last_undegraded":"2023-10-14T01:10:00.210168+0000","last_fullsized":"2023-10-14T01:10:00.210168+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:36:13.665649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'457146","reported_seq":910964,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.855510+0000","last_change":"2023-10-14T00:04:55.942378+0000","last_active":"2023-10-14T01:10:07.855510+0000","last_peered":"2023-10-14T01:10:07.855510+0000","last_clean":"2023-10-14T01:10:07.855510+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:10:07.855510+0000","last_undegraded":"2023-10-14T01:10:07.855510+0000","last_fullsized":"2023-10-14T01:10:07.855510+0000","mapping_epoch":15,"log_start":"84'447133","ondisk_log_start":"84'447133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:31:35.598409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7117701139,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":456250,"num_read_kb":1794768,"num_write":454574,"num_write_kb":1818290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'12429","reported_seq":17028,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:07.780555+0000","last_change":"2023-10-14T00:04:47.888394+0000","last_active":"2023-10-14T01:10:07.780555+0000","last_peered":"2023-10-14T01:10:07.780555+0000","last_clean":"2023-10-14T01:10:07.780555+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:10:07.780555+0000","last_undegraded":"2023-10-14T01:10:07.780555+0000","last_fullsized":"2023-10-14T01:10:07.780555+0000","mapping_epoch":12,"log_start":"65'2400","ondisk_log_start":"65'2400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:15:14.957936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":244,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-14T01:09:07.498152+0000","last_change":"2023-10-14T00:04:42.867239+0000","last_active":"2023-10-14T01:09:07.498152+0000","last_peered":"2023-10-14T01:09:07.498152+0000","last_clean":"2023-10-14T01:09:07.498152+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:09:07.498152+0000","last_undegraded":"2023-10-14T01:09:07.498152+0000","last_fullsized":"2023-10-14T01:09:07.498152+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_clean_scrub_stamp":"2023-10-14T00:04:41.840847+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:45:11.022601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28349968403,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1816771,"num_read_kb":7146956,"num_write":1813163,"num_write_kb":7252646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14853423104,"data_stored":17921450022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40192,"ondisk_log_size":40192,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45246,"ondisk_log_size":45246,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8684532,"kb_used_data":7251624,"kb_used_omap":976,"kb_used_meta":1431919,"kb_avail":85687308,"statfs":{"total":96636764160,"available":87743803392,"internally_reserved":0,"allocated":7425662976,"data_stored":8959192746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1466285341},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.33900000000000002,"15min":0.40100000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.747,"5min":0.99199999999999999,"15min":4.6189999999999998},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.34699999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":3.4950000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.23300000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.42299999999999999,"15min":0.49399999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":1.2330000000000001,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.46400000000000002,"15min":0.51100000000000001},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":1.2629999999999999,"5min":4.2060000000000004,"15min":16.747},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739158,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5454340,"kb_used_data":3637628,"kb_used_omap":510,"kb_used_meta":1816193,"kb_avail":88917500,"statfs":{"total":96636764160,"available":91051520000,"internally_reserved":0,"allocated":3724931072,"data_stored":4492040745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1859781716},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,6,2,8,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":3.0209999999999999,"5min":4.117,"15min":5.2370000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.56499999999999995,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":0.46800000000000003,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":4.4740000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.623,"5min":5.6040000000000001,"15min":5.6040000000000001},"last":0.252},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42299999999999999,"15min":0.47299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.65300000000000002,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.27900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739160,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5442828,"kb_used_data":3617540,"kb_used_omap":502,"kb_used_meta":1824777,"kb_avail":88929012,"statfs":{"total":96636764160,"available":91063308288,"internally_reserved":0,"allocated":3704360960,"data_stored":4469921372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1868572118},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,0,4,6,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.40000000000000002,"15min":0.41799999999999998},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":4.452,"5min":4.452,"15min":8.3239999999999998},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.33800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.66700000000000004,"15min":0.56499999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.188},"max":{"1min":4.5449999999999999,"5min":5.875,"15min":26.074999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":4.492,"5min":4.492,"15min":6.5449999999999999},"last":0.26300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3703341056,"data_stored":4469526528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3723382784,"data_stored":4491210771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7426699264,"data_stored":8960712723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:10:11.640 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T01:10:12.225 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:12.226 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":85,"fsid":"0234826a-19f7-4f4f-8da0-4e6264e0bcb3","created":"2023-10-14T00:04:30.860625+0000","modified":"2023-10-14T01:09:17.019944+0000","last_up_change":"2023-10-14T00:04:40.716111+0000","last_in_change":"2023-10-14T00:04:31.781133+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T00:04:41.634304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-14T00:04:46.262162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2023-10-14T00:04:52.060198+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":63,"snap_epoch":61,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"16ee973b-0431-49dd-8db9-d15a5f4d04fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6801","nonce":2607237379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6803","nonce":2607237379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6807","nonce":2607237379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":2607237379},{"type":"v1","addr":"172.21.15.105:6805","nonce":2607237379}]},"public_addr":"172.21.15.105:6801/2607237379","cluster_addr":"172.21.15.105:6803/2607237379","heartbeat_back_addr":"172.21.15.105:6807/2607237379","heartbeat_front_addr":"172.21.15.105:6805/2607237379","state":["exists","up"]},{"osd":1,"uuid":"3b8038c1-9263-4c09-8d87-77b387bb5668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6809","nonce":2843110786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6811","nonce":2843110786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6815","nonce":2843110786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":2843110786},{"type":"v1","addr":"172.21.15.105:6813","nonce":2843110786}]},"public_addr":"172.21.15.105:6809/2843110786","cluster_addr":"172.21.15.105:6811/2843110786","heartbeat_back_addr":"172.21.15.105:6815/2843110786","heartbeat_front_addr":"172.21.15.105:6813/2843110786","state":["exists","up"]},{"osd":2,"uuid":"3b32ac1b-c76d-4d44-b0fb-98ec6ceb174f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6817","nonce":4190342661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6819","nonce":4190342661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6823","nonce":4190342661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4190342661},{"type":"v1","addr":"172.21.15.105:6821","nonce":4190342661}]},"public_addr":"172.21.15.105:6817/4190342661","cluster_addr":"172.21.15.105:6819/4190342661","heartbeat_back_addr":"172.21.15.105:6823/4190342661","heartbeat_front_addr":"172.21.15.105:6821/4190342661","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.250185+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.382036+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T00:04:37.014142+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/1796457399":"2023-10-14T01:25:44.299376+0000","172.21.15.105:0/4114602057":"2023-10-14T01:22:06.395992+0000","172.21.15.105:0/3712341576":"2023-10-14T01:24:35.222129+0000","172.21.15.105:0/995254717":"2023-10-14T01:20:55.173529+0000","172.21.15.105:0/2655824907":"2023-10-14T01:15:11.160896+0000","172.21.15.105:0/654405172":"2023-10-14T01:23:26.572550+0000","172.21.15.105:0/390132398":"2023-10-14T01:13:51.090065+0000","172.21.15.105:0/1238328416":"2023-10-14T01:10:26.279568+0000","172.21.15.105:0/396685582":"2023-10-14T01:11:34.890884+0000","172.21.15.105:0/3267454688":"2023-10-14T01:12:42.952128+0000","172.21.15.105:0/3372654654":"2023-10-14T01:18:37.783047+0000","172.21.15.105:0/3800667290":"2023-10-14T01:17:29.177833+0000","172.21.15.105:0/3438481267":"2023-10-14T01:16:20.419977+0000","172.21.15.105:0/115563815":"2023-10-14T01:19:46.484357+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T01:10:13.253 INFO:tasks.ceph:Scrubbing osd.0 2023-10-14T01:10:13.253 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:10:13.527 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2023-10-14T01:10:13.527 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:10:13.528 INFO:teuthology.orchestra.run.smithi105.stdout:} 2023-10-14T01:10:13.563 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-14T01:10:14.147 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 0 to deep-scrub 2023-10-14T01:10:14.178 INFO:tasks.ceph:Scrubbing osd.1 2023-10-14T01:10:14.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:10:14.458 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2023-10-14T01:10:14.458 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:10:14.459 INFO:teuthology.orchestra.run.smithi105.stdout:} 2023-10-14T01:10:14.492 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-14T01:10:14.991 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 1 to deep-scrub 2023-10-14T01:10:15.035 INFO:tasks.ceph:Scrubbing osd.2 2023-10-14T01:10:15.036 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-14T01:10:15.295 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2023-10-14T01:10:15.295 INFO:teuthology.orchestra.run.smithi105.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T01:10:15.295 INFO:teuthology.orchestra.run.smithi105.stdout:} 2023-10-14T01:10:15.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-14T01:10:15.933 INFO:teuthology.orchestra.run.smithi105.stderr:instructed osd(s) 2 to deep-scrub 2023-10-14T01:10:15.982 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:16.583 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:16.584 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:10:16.609 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2051,"stamp":"2023-10-14T01:10:16.343273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28359479116,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1841387,"num_read_kb":7174550,"num_write":1816128,"num_write_kb":7263523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85538,"ondisk_log_size":85538,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19669184,"kb_used_data":14523496,"kb_used_omap":1990,"kb_used_meta":5143673,"kb_avail":263446336,"statfs":{"total":289910292480,"available":269769048064,"internally_reserved":0,"allocated":14872059904,"data_stored":17938259759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5267121991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,3,6,18,36,6,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18165760,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4520,"num_read_kb":18020,"num_write":4429,"num_write_kb":17716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002168"},"pg_stats":[{"pgid":"2.7","version":"85'9421","reported_seq":13081,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:12.804097+0000","last_change":"2023-10-14T00:04:47.892979+0000","last_active":"2023-10-14T01:10:12.804097+0000","last_peered":"2023-10-14T01:10:12.804097+0000","last_clean":"2023-10-14T01:10:12.804097+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:10:12.804097+0000","last_undegraded":"2023-10-14T01:10:12.804097+0000","last_fullsized":"2023-10-14T01:10:12.804097+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9421,"log_dups_size":0,"ondisk_log_size":9421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T00:19:15.695751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"85'6339","reported_seq":9726,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:12.780015+0000","last_change":"2023-10-14T00:04:47.892672+0000","last_active":"2023-10-14T01:10:12.780015+0000","last_peered":"2023-10-14T01:10:12.780015+0000","last_clean":"2023-10-14T01:10:12.780015+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:10:12.780015+0000","last_undegraded":"2023-10-14T01:10:12.780015+0000","last_fullsized":"2023-10-14T01:10:12.780015+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":6339,"log_dups_size":0,"ondisk_log_size":6339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:59:03.121676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"61'12","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:09:17.040477+0000","last_change":"2023-10-14T00:04:47.893288+0000","last_active":"2023-10-14T01:09:17.040477+0000","last_peered":"2023-10-14T01:09:17.040477+0000","last_clean":"2023-10-14T01:09:17.040477+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:09:17.040477+0000","last_undegraded":"2023-10-14T01:09:17.040477+0000","last_fullsized":"2023-10-14T01:09:17.040477+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:05:35.203825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"85'12519","reported_seq":17770,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:12.808866+0000","last_change":"2023-10-14T00:04:47.892782+0000","last_active":"2023-10-14T01:10:12.808866+0000","last_peered":"2023-10-14T01:10:12.808866+0000","last_clean":"2023-10-14T01:10:12.808866+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:10:12.808866+0000","last_undegraded":"2023-10-14T01:10:12.808866+0000","last_fullsized":"2023-10-14T01:10:12.808866+0000","mapping_epoch":12,"log_start":"65'2500","ondisk_log_start":"65'2500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:09:57.647677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"85'450080","reported_seq":895944,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:11.827736+0000","last_change":"2023-10-14T00:04:55.935661+0000","last_active":"2023-10-14T01:10:11.827736+0000","last_peered":"2023-10-14T01:10:11.827736+0000","last_clean":"2023-10-14T01:10:11.827736+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:10:11.827736+0000","last_undegraded":"2023-10-14T01:10:11.827736+0000","last_fullsized":"2023-10-14T01:10:11.827736+0000","mapping_epoch":15,"log_start":"84'440029","ondisk_log_start":"84'440029","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:11:39.347355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":6985244672,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":448246,"num_read_kb":1763320,"num_write":447558,"num_write_kb":1790232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"85'9268","reported_seq":12813,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:10.454110+0000","last_change":"2023-10-14T00:04:47.890020+0000","last_active":"2023-10-14T01:10:10.454110+0000","last_peered":"2023-10-14T01:10:10.454110+0000","last_clean":"2023-10-14T01:10:10.454110+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:10:10.454110+0000","last_undegraded":"2023-10-14T01:10:10.454110+0000","last_fullsized":"2023-10-14T01:10:10.454110+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":9268,"log_dups_size":0,"ondisk_log_size":9268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T03:42:10.640763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"85'457568","reported_seq":910106,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:12.888764+0000","last_change":"2023-10-14T00:04:55.940744+0000","last_active":"2023-10-14T01:10:12.888764+0000","last_peered":"2023-10-14T01:10:12.888764+0000","last_clean":"2023-10-14T01:10:12.888764+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:10:12.888764+0000","last_undegraded":"2023-10-14T01:10:12.888764+0000","last_fullsized":"2023-10-14T01:10:12.888764+0000","mapping_epoch":15,"log_start":"84'447525","ondisk_log_start":"84'447525","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:03:23.735983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7115710464,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":454966,"num_read_kb":1789428,"num_write":454998,"num_write_kb":1819992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"84'139","reported_seq":3649,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:10.557795+0000","last_change":"2023-10-14T00:04:47.889952+0000","last_active":"2023-10-14T01:10:10.557795+0000","last_peered":"2023-10-14T01:10:10.557795+0000","last_clean":"2023-10-14T01:10:10.557795+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:10:10.557795+0000","last_undegraded":"2023-10-14T01:10:10.557795+0000","last_fullsized":"2023-10-14T01:10:10.557795+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:33:09.537307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"85'460239","reported_seq":916720,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:11.805212+0000","last_change":"2023-10-14T00:04:55.935824+0000","last_active":"2023-10-14T01:10:11.805212+0000","last_peered":"2023-10-14T01:10:11.805212+0000","last_clean":"2023-10-14T01:10:11.805212+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:10:11.805212+0000","last_undegraded":"2023-10-14T01:10:11.805212+0000","last_fullsized":"2023-10-14T01:10:11.805212+0000","mapping_epoch":15,"log_start":"84'450224","ondisk_log_start":"84'450224","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:29:57.244648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7138095104,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":458918,"num_read_kb":1805856,"num_write":457663,"num_write_kb":1830652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"84'55","reported_seq":1309,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:05.331478+0000","last_change":"2023-10-14T00:04:47.890489+0000","last_active":"2023-10-14T01:10:05.331478+0000","last_peered":"2023-10-14T01:10:05.331478+0000","last_clean":"2023-10-14T01:10:05.331478+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:10:05.331478+0000","last_undegraded":"2023-10-14T01:10:05.331478+0000","last_fullsized":"2023-10-14T01:10:05.331478+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:36:13.665649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"85'457671","reported_seq":912060,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:12.894588+0000","last_change":"2023-10-14T00:04:55.942378+0000","last_active":"2023-10-14T01:10:12.894588+0000","last_peered":"2023-10-14T01:10:12.894588+0000","last_clean":"2023-10-14T01:10:12.894588+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:10:12.894588+0000","last_undegraded":"2023-10-14T01:10:12.894588+0000","last_fullsized":"2023-10-14T01:10:12.894588+0000","mapping_epoch":15,"log_start":"84'447633","ondisk_log_start":"84'447633","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:31:35.598409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7119835155,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":456820,"num_read_kb":1797032,"num_write":455099,"num_write_kb":1820390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'12441","reported_seq":17044,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:12.835656+0000","last_change":"2023-10-14T00:04:47.888394+0000","last_active":"2023-10-14T01:10:12.835656+0000","last_peered":"2023-10-14T01:10:12.835656+0000","last_clean":"2023-10-14T01:10:12.835656+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:10:12.835656+0000","last_undegraded":"2023-10-14T01:10:12.835656+0000","last_fullsized":"2023-10-14T01:10:12.835656+0000","mapping_epoch":12,"log_start":"65'2400","ondisk_log_start":"65'2400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:46.870589+0000","last_clean_scrub_stamp":"2023-10-14T00:04:46.870589+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T08:15:14.957936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":244,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-14T01:09:07.498152+0000","last_change":"2023-10-14T00:04:42.867239+0000","last_active":"2023-10-14T01:09:07.498152+0000","last_peered":"2023-10-14T01:09:07.498152+0000","last_clean":"2023-10-14T01:09:07.498152+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:09:07.498152+0000","last_undegraded":"2023-10-14T01:09:07.498152+0000","last_fullsized":"2023-10-14T01:09:07.498152+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:41.840847+0000","last_clean_scrub_stamp":"2023-10-14T00:04:41.840847+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:45:11.022601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28358885395,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1818950,"num_read_kb":7155636,"num_write":1815318,"num_write_kb":7261266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14870970368,"data_stored":17938997286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45294,"ondisk_log_size":45294,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8737848,"kb_used_data":7260528,"kb_used_omap":976,"kb_used_meta":1476335,"kb_avail":85633992,"statfs":{"total":96636764160,"available":87689207808,"internally_reserved":0,"allocated":7434780672,"data_stored":8968310442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1511767325},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.33900000000000002,"15min":0.40100000000000002},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.747,"5min":0.99199999999999999,"15min":4.6189999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41499999999999998,"15min":0.34699999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":3.4950000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.22}]},{"osd":1,"last update":"Sat Oct 14 01:09:17 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.42299999999999999,"15min":0.49399999999999999},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":1.2330000000000001,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.318},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.46400000000000002,"15min":0.51100000000000001},"min":{"1min":0.252,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":1.2629999999999999,"5min":4.2060000000000004,"15min":16.747},"last":0.34300000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5462020,"kb_used_data":3641148,"kb_used_omap":510,"kb_used_meta":1820353,"kb_avail":88909820,"statfs":{"total":96636764160,"available":91043655680,"internally_reserved":0,"allocated":3728535552,"data_stored":4495645225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1864041556},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,8,29,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":3.0209999999999999,"5min":4.117,"15min":5.2370000000000001},"last":0.314},{"interface":"front","average":{"1min":0.314,"5min":0.56499999999999995,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":0.46800000000000003,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.27000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.623,"5min":5.6040000000000001,"15min":5.6040000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42299999999999999,"15min":0.47299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.65300000000000002,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739161,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5469316,"kb_used_data":3621820,"kb_used_omap":502,"kb_used_meta":1846985,"kb_avail":88902524,"statfs":{"total":96636764160,"available":91036184576,"internally_reserved":0,"allocated":3708743680,"data_stored":4474304092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1891313110},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,8,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.40000000000000002,"15min":0.41799999999999998},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":4.452,"5min":4.452,"15min":8.3239999999999998},"last":1.8109999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.215}]},{"osd":2,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.66700000000000004,"15min":0.56499999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.188},"max":{"1min":4.5449999999999999,"5min":5.875,"15min":26.074999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":4.492,"5min":4.492,"15min":6.5449999999999999},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3707789312,"data_stored":4473974784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3727769600,"data_stored":4495597587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7435411456,"data_stored":8969424915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:10:16.612 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.613 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.613 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.613 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.614 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.614 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.615 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.615 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.615 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.616 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.616 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.617 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-14T00:04:46.870589+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=46, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.617 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-14T00:04:41.840847+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=41, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:16.617 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:10:36.618 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:37.092 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:37.093 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:10:37.139 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2062,"stamp":"2023-10-14T01:10:36.346652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28391366476,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1849172,"num_read_kb":7205542,"num_write":1823945,"num_write_kb":7294791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85781,"ondisk_log_size":85781,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19925532,"kb_used_data":14588100,"kb_used_omap":1990,"kb_used_meta":5335417,"kb_avail":263189988,"statfs":{"total":289910292480,"available":269506547712,"internally_reserved":0,"allocated":14938214400,"data_stored":18004401955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5463467847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,10,12,7,11,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15388672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3787,"num_read_kb":15096,"num_write":3776,"num_write_kb":15104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5145,"num_omap_keys":84,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001766"},"pg_stats":[{"pgid":"2.7","version":"86'9447","reported_seq":13129,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:30.934126+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:10:30.934126+0000","last_peered":"2023-10-14T01:10:30.934126+0000","last_clean":"2023-10-14T01:10:30.934126+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:10:30.934126+0000","last_undegraded":"2023-10-14T01:10:30.934126+0000","last_fullsized":"2023-10-14T01:10:30.934126+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6364","reported_seq":9769,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:30.933467+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:10:30.933467+0000","last_peered":"2023-10-14T01:10:30.933467+0000","last_clean":"2023-10-14T01:10:30.933467+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:10:30.933467+0000","last_undegraded":"2023-10-14T01:10:30.933467+0000","last_fullsized":"2023-10-14T01:10:30.933467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:21.898506+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:10:21.898506+0000","last_peered":"2023-10-14T01:10:21.898506+0000","last_clean":"2023-10-14T01:10:21.898506+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:10:21.898506+0000","last_undegraded":"2023-10-14T01:10:21.898506+0000","last_fullsized":"2023-10-14T01:10:21.898506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'12559","reported_seq":17836,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:30.934104+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:10:30.934104+0000","last_peered":"2023-10-14T01:10:30.934104+0000","last_clean":"2023-10-14T01:10:30.934104+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:10:30.934104+0000","last_undegraded":"2023-10-14T01:10:30.934104+0000","last_fullsized":"2023-10-14T01:10:30.934104+0000","mapping_epoch":12,"log_start":"65'2500","ondisk_log_start":"65'2500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'452106","reported_seq":899849,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:31.891699+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:10:31.891699+0000","last_peered":"2023-10-14T01:10:31.891699+0000","last_clean":"2023-10-14T01:10:31.891699+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:10:31.891699+0000","last_undegraded":"2023-10-14T01:10:31.891699+0000","last_fullsized":"2023-10-14T01:10:31.891699+0000","mapping_epoch":15,"log_start":"84'442029","ondisk_log_start":"84'442029","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":60,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 8s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":6993305600,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":450115,"num_read_kb":1770764,"num_write":449544,"num_write_kb":1798176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'9315","reported_seq":12882,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:30.934005+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:10:30.934005+0000","last_peered":"2023-10-14T01:10:30.934005+0000","last_clean":"2023-10-14T01:10:30.934005+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:10:30.934005+0000","last_undegraded":"2023-10-14T01:10:30.934005+0000","last_fullsized":"2023-10-14T01:10:30.934005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9315,"log_dups_size":0,"ondisk_log_size":9315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'459538","reported_seq":913981,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:33.014121+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:10:33.014121+0000","last_peered":"2023-10-14T01:10:33.014121+0000","last_clean":"2023-10-14T01:10:33.014121+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:10:33.014121+0000","last_undegraded":"2023-10-14T01:10:33.014121+0000","last_fullsized":"2023-10-14T01:10:33.014121+0000","mapping_epoch":15,"log_start":"84'449525","ondisk_log_start":"84'449525","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":37,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7123599360,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":456864,"num_read_kb":1797004,"num_write":456941,"num_write_kb":1827764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"85'146","reported_seq":3678,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:30.947772+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:10:30.947772+0000","last_peered":"2023-10-14T01:10:30.947772+0000","last_clean":"2023-10-14T01:10:30.947772+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:10:30.947772+0000","last_undegraded":"2023-10-14T01:10:30.947772+0000","last_fullsized":"2023-10-14T01:10:30.947772+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'462251","reported_seq":920718,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:31.885428+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:10:31.885428+0000","last_peered":"2023-10-14T01:10:31.885428+0000","last_clean":"2023-10-14T01:10:31.885428+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:10:31.885428+0000","last_undegraded":"2023-10-14T01:10:31.885428+0000","last_fullsized":"2023-10-14T01:10:31.885428+0000","mapping_epoch":15,"log_start":"84'452224","ondisk_log_start":"84'452224","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":82,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 12s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7146168320,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":460891,"num_read_kb":1813700,"num_write":459619,"num_write_kb":1838476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"85'68","reported_seq":1336,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:27.061145+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:10:27.061145+0000","last_peered":"2023-10-14T01:10:27.061145+0000","last_clean":"2023-10-14T01:10:27.061145+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:10:27.061145+0000","last_undegraded":"2023-10-14T01:10:27.061145+0000","last_fullsized":"2023-10-14T01:10:27.061145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'459603","reported_seq":916039,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:33.024695+0000","last_change":"2023-10-14T01:10:32.948847+0000","last_active":"2023-10-14T01:10:33.024695+0000","last_peered":"2023-10-14T01:10:33.024695+0000","last_clean":"2023-10-14T01:10:33.024695+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:10:33.024695+0000","last_undegraded":"2023-10-14T01:10:33.024695+0000","last_fullsized":"2023-10-14T01:10:33.024695+0000","mapping_epoch":15,"log_start":"84'449533","ondisk_log_start":"84'449533","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7127699475,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":458865,"num_read_kb":1805160,"num_write":457031,"num_write_kb":1828118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'12472","reported_seq":17101,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:30.934825+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:10:30.934825+0000","last_peered":"2023-10-14T01:10:30.934825+0000","last_clean":"2023-10-14T01:10:30.934825+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:10:30.934825+0000","last_undegraded":"2023-10-14T01:10:30.934825+0000","last_fullsized":"2023-10-14T01:10:30.934825+0000","mapping_epoch":12,"log_start":"65'2400","ondisk_log_start":"65'2400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:16.545738+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:10:16.545738+0000","last_peered":"2023-10-14T01:10:16.545738+0000","last_clean":"2023-10-14T01:10:16.545738+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:10:16.545738+0000","last_undegraded":"2023-10-14T01:10:16.545738+0000","last_fullsized":"2023-10-14T01:10:16.545738+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28390772755,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1826735,"num_read_kb":7186628,"num_write":1823135,"num_write_kb":7292534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14935031808,"data_stored":18003058726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40187,"ondisk_log_size":40187,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45495,"ondisk_log_size":45495,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739162,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8866032,"kb_used_data":7293160,"kb_used_omap":976,"kb_used_meta":1571887,"kb_avail":85505808,"statfs":{"total":96636764160,"available":87557947392,"internally_reserved":0,"allocated":7468195840,"data_stored":9001721510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1609612573},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.35399999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.492,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.215}]},{"osd":1,"last update":"Sat Oct 14 01:10:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":4.1740000000000004,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.51100000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739163,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5538312,"kb_used_data":3657344,"kb_used_omap":510,"kb_used_meta":1880449,"kb_avail":88833528,"statfs":{"total":96636764160,"available":90965532672,"internally_reserved":0,"allocated":3745120256,"data_stored":4512225829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1925579860},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,7,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":3.0209999999999999,"5min":4.117,"15min":5.2370000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.56499999999999995,"15min":0.47399999999999998},"min":{"1min":0.20499999999999999,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":0.46800000000000003,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.221}]},{"osd":2,"last update":"Sat Oct 14 01:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.379,"15min":0.42799999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.623,"5min":5.6040000000000001,"15min":5.6040000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42299999999999999,"15min":0.47299999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.65300000000000002,"5min":5.8529999999999998,"15min":5.8529999999999998},"last":0.74299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739165,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5521188,"kb_used_data":3637596,"kb_used_omap":502,"kb_used_meta":1883081,"kb_avail":88850652,"statfs":{"total":96636764160,"available":90983067648,"internally_reserved":0,"allocated":3724898304,"data_stored":4490454616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1928275414},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,5,11,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.40000000000000002,"15min":0.41799999999999998},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":4.452,"5min":4.452,"15min":8.3239999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":4.6420000000000003,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.318}]},{"osd":2,"last update":"Sat Oct 14 01:09:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.66700000000000004,"15min":0.56499999999999995},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.188},"max":{"1min":4.5449999999999999,"5min":5.875,"15min":26.074999999999999},"last":4.3390000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.41699999999999998},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":4.492,"5min":4.492,"15min":6.5449999999999999},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3723939840,"data_stored":4490125312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3743629312,"data_stored":4511457299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7467462656,"data_stored":9001476115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:10:37.141 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:37.142 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:37.142 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:37.142 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:37.143 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:10:57.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:10:57.687 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:10:57.689 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:10:57.718 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2072,"stamp":"2023-10-14T01:10:56.350186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28421873484,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1856837,"num_read_kb":7236078,"num_write":1831406,"num_write_kb":7324635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85883,"ondisk_log_size":85883,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20111256,"kb_used_data":14647552,"kb_used_omap":1990,"kb_used_meta":5461689,"kb_avail":263004264,"statfs":{"total":289910292480,"available":269316366336,"internally_reserved":0,"allocated":14999093248,"data_stored":18065280803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5592770375},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,12,14,12,15,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15314944,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3949,"num_read_kb":15720,"num_write":3758,"num_write_kb":15032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002307"},"pg_stats":[{"pgid":"2.7","version":"86'9516","reported_seq":13208,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:52.005453+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:10:52.005453+0000","last_peered":"2023-10-14T01:10:52.005453+0000","last_clean":"2023-10-14T01:10:52.005453+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:10:52.005453+0000","last_undegraded":"2023-10-14T01:10:52.005453+0000","last_fullsized":"2023-10-14T01:10:52.005453+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9516,"log_dups_size":0,"ondisk_log_size":9516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6410","reported_seq":9823,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:52.000539+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:10:52.000539+0000","last_peered":"2023-10-14T01:10:52.000539+0000","last_clean":"2023-10-14T01:10:52.000539+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:10:52.000539+0000","last_undegraded":"2023-10-14T01:10:52.000539+0000","last_fullsized":"2023-10-14T01:10:52.000539+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6410,"log_dups_size":0,"ondisk_log_size":6410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:21.898506+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:10:21.898506+0000","last_peered":"2023-10-14T01:10:21.898506+0000","last_clean":"2023-10-14T01:10:21.898506+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:10:21.898506+0000","last_undegraded":"2023-10-14T01:10:21.898506+0000","last_fullsized":"2023-10-14T01:10:21.898506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'12651","reported_seq":17943,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:52.018999+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:10:52.018999+0000","last_peered":"2023-10-14T01:10:52.018999+0000","last_clean":"2023-10-14T01:10:52.018999+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:10:52.018999+0000","last_undegraded":"2023-10-14T01:10:52.018999+0000","last_fullsized":"2023-10-14T01:10:52.018999+0000","mapping_epoch":12,"log_start":"65'2600","ondisk_log_start":"65'2600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'454032","reported_seq":903656,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:52.020005+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:10:52.020005+0000","last_peered":"2023-10-14T01:10:52.020005+0000","last_clean":"2023-10-14T01:10:52.020005+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:10:52.020005+0000","last_undegraded":"2023-10-14T01:10:52.020005+0000","last_fullsized":"2023-10-14T01:10:52.020005+0000","mapping_epoch":15,"log_start":"84'444029","ondisk_log_start":"84'444029","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":196,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 28s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7000727552,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":451985,"num_read_kb":1778228,"num_write":451371,"num_write_kb":1805484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'9366","reported_seq":12945,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:51.413724+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:10:51.413724+0000","last_peered":"2023-10-14T01:10:51.413724+0000","last_clean":"2023-10-14T01:10:51.413724+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:10:51.413724+0000","last_undegraded":"2023-10-14T01:10:51.413724+0000","last_fullsized":"2023-10-14T01:10:51.413724+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'461461","reported_seq":917813,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:53.157517+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:10:53.157517+0000","last_peered":"2023-10-14T01:10:53.157517+0000","last_clean":"2023-10-14T01:10:53.157517+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:10:53.157517+0000","last_undegraded":"2023-10-14T01:10:53.157517+0000","last_fullsized":"2023-10-14T01:10:53.157517+0000","mapping_epoch":15,"log_start":"85'451425","ondisk_log_start":"85'451425","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":178,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 25s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7131086848,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":458763,"num_read_kb":1804548,"num_write":458766,"num_write_kb":1835064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"85'146","reported_seq":3690,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:51.435502+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:10:51.435502+0000","last_peered":"2023-10-14T01:10:51.435502+0000","last_clean":"2023-10-14T01:10:51.435502+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:10:51.435502+0000","last_undegraded":"2023-10-14T01:10:51.435502+0000","last_fullsized":"2023-10-14T01:10:51.435502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'464228","reported_seq":924654,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:10:52.011700+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:10:52.011700+0000","last_peered":"2023-10-14T01:10:52.011700+0000","last_clean":"2023-10-14T01:10:52.011700+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:10:52.011700+0000","last_undegraded":"2023-10-14T01:10:52.011700+0000","last_fullsized":"2023-10-14T01:10:52.011700+0000","mapping_epoch":15,"log_start":"85'454224","ondisk_log_start":"85'454224","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":220,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 32s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7153885184,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":462838,"num_read_kb":1821444,"num_write":461502,"num_write_kb":1846008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"85'68","reported_seq":1340,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:46.294825+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:10:46.294825+0000","last_peered":"2023-10-14T01:10:46.294825+0000","last_clean":"2023-10-14T01:10:46.294825+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:10:46.294825+0000","last_undegraded":"2023-10-14T01:10:46.294825+0000","last_fullsized":"2023-10-14T01:10:46.294825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'461529","reported_seq":919918,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:53.174816+0000","last_change":"2023-10-14T01:10:45.000612+0000","last_active":"2023-10-14T01:10:53.174816+0000","last_peered":"2023-10-14T01:10:53.174816+0000","last_clean":"2023-10-14T01:10:53.174816+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:10:53.174816+0000","last_undegraded":"2023-10-14T01:10:53.174816+0000","last_fullsized":"2023-10-14T01:10:53.174816+0000","mapping_epoch":15,"log_start":"84'451433","ondisk_log_start":"84'451433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7135580179,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":460814,"num_read_kb":1812944,"num_write":458957,"num_write_kb":1835822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'12564","reported_seq":17206,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:52.044288+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:10:52.044288+0000","last_peered":"2023-10-14T01:10:52.044288+0000","last_clean":"2023-10-14T01:10:52.044288+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:10:52.044288+0000","last_undegraded":"2023-10-14T01:10:52.044288+0000","last_fullsized":"2023-10-14T01:10:52.044288+0000","mapping_epoch":12,"log_start":"65'2500","ondisk_log_start":"65'2500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:16.545738+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:10:16.545738+0000","last_peered":"2023-10-14T01:10:16.545738+0000","last_clean":"2023-10-14T01:10:16.545738+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:10:16.545738+0000","last_undegraded":"2023-10-14T01:10:16.545738+0000","last_fullsized":"2023-10-14T01:10:16.545738+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28421279763,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1834400,"num_read_kb":7217164,"num_write":1830596,"num_write_kb":7322378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14996185088,"data_stored":18064212006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40139,"ondisk_log_size":40139,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45645,"ondisk_log_size":45645,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739166,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8967724,"kb_used_data":7322724,"kb_used_omap":976,"kb_used_meta":1644015,"kb_avail":85404116,"statfs":{"total":96636764160,"available":87453814784,"internally_reserved":0,"allocated":7498469376,"data_stored":9031995046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1683471645},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.35399999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.492,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.23999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:10:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":4.1740000000000004,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.315}]}]},{"osd":1,"up_from":8,"seq":34359739167,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5571296,"kb_used_data":3672280,"kb_used_omap":510,"kb_used_meta":1898497,"kb_avail":88800544,"statfs":{"total":96636764160,"available":90931757056,"internally_reserved":0,"allocated":3760414720,"data_stored":4527520293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1944061012},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.47799999999999998,"15min":0.46800000000000003},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.77300000000000002,"5min":4.117,"15min":5.2370000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.46600000000000003},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":4.4740000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.22}]},{"osd":2,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.42299999999999999},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.69299999999999995,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.46800000000000003},"min":{"1min":0.25800000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.74299999999999999,"5min":1.48,"15min":5.8529999999999998},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739169,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5572236,"kb_used_data":3652548,"kb_used_omap":502,"kb_used_meta":1919177,"kb_avail":88799604,"statfs":{"total":96636764160,"available":90930794496,"internally_reserved":0,"allocated":3740209152,"data_stored":4505765464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1965237718},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,10,7,3,12,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.40600000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":1.8109999999999999,"5min":4.452,"15min":8.3239999999999998},"last":0.375},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.497,"15min":0.41999999999999998},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":3.2709999999999999,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.59099999999999997},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.188},"max":{"1min":4.3390000000000004,"5min":5.875,"15min":26.074999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.47699999999999998,"15min":0.41699999999999998},"min":{"1min":0.255,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.78400000000000003,"5min":4.492,"15min":6.5449999999999999},"last":0.55300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3739144192,"data_stored":4505329664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3758989312,"data_stored":4526817299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7498051584,"data_stored":9032065043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:10:57.720 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:57.721 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:57.721 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:57.722 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:10:57.722 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:10.866 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:11:10.867+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:11:14.524 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:14.515+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:11:14.524 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:14.515+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:11:14.524 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:14.515+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:11:14.531 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:14.523+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:11:14.531 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:14.523+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:11:17.723 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:18.285 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:11:18.285 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:11:18.317 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2082,"stamp":"2023-10-14T01:11:16.353459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28452704076,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1864530,"num_read_kb":7266750,"num_write":1838965,"num_write_kb":7354871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86034,"ondisk_log_size":86034,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20345908,"kb_used_data":14707356,"kb_used_omap":1990,"kb_used_meta":5636537,"kb_avail":262769612,"statfs":{"total":289910292480,"available":269076082688,"internally_reserved":0,"allocated":15060332544,"data_stored":18126520099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":5771814727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,6,14,6,19,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15409152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3903,"num_read_kb":15564,"num_write":3785,"num_write_kb":15140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001927"},"pg_stats":[{"pgid":"2.7","version":"86'9549","reported_seq":13253,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:12.808391+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:11:12.808391+0000","last_peered":"2023-10-14T01:11:12.808391+0000","last_clean":"2023-10-14T01:11:12.808391+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:11:12.808391+0000","last_undegraded":"2023-10-14T01:11:12.808391+0000","last_fullsized":"2023-10-14T01:11:12.808391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9549,"log_dups_size":0,"ondisk_log_size":9549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6432","reported_seq":9853,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:12.800057+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:11:12.800057+0000","last_peered":"2023-10-14T01:11:12.800057+0000","last_clean":"2023-10-14T01:11:12.800057+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:11:12.800057+0000","last_undegraded":"2023-10-14T01:11:12.800057+0000","last_fullsized":"2023-10-14T01:11:12.800057+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6432,"log_dups_size":0,"ondisk_log_size":6432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:21.898506+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:10:21.898506+0000","last_peered":"2023-10-14T01:10:21.898506+0000","last_clean":"2023-10-14T01:10:21.898506+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:10:21.898506+0000","last_undegraded":"2023-10-14T01:10:21.898506+0000","last_fullsized":"2023-10-14T01:10:21.898506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'12695","reported_seq":18003,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:12.814915+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:11:12.814915+0000","last_peered":"2023-10-14T01:11:12.814915+0000","last_clean":"2023-10-14T01:11:12.814915+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:11:12.814915+0000","last_undegraded":"2023-10-14T01:11:12.814915+0000","last_fullsized":"2023-10-14T01:11:12.814915+0000","mapping_epoch":12,"log_start":"65'2600","ondisk_log_start":"65'2600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"86'456005","reported_seq":907517,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:12.149657+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:11:12.149657+0000","last_peered":"2023-10-14T01:11:12.149657+0000","last_clean":"2023-10-14T01:11:12.149657+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:11:12.149657+0000","last_undegraded":"2023-10-14T01:11:12.149657+0000","last_fullsized":"2023-10-14T01:11:12.149657+0000","mapping_epoch":15,"log_start":"85'445929","ondisk_log_start":"85'445929","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":367,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 48s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7008329728,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":453857,"num_read_kb":1785688,"num_write":453226,"num_write_kb":1812904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'9412","reported_seq":13003,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:11.893628+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:11:11.893628+0000","last_peered":"2023-10-14T01:11:11.893628+0000","last_clean":"2023-10-14T01:11:11.893628+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:11:11.893628+0000","last_undegraded":"2023-10-14T01:11:11.893628+0000","last_fullsized":"2023-10-14T01:11:11.893628+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9412,"log_dups_size":0,"ondisk_log_size":9412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'463428","reported_seq":921706,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:13.340161+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:11:13.340161+0000","last_peered":"2023-10-14T01:11:13.340161+0000","last_clean":"2023-10-14T01:11:13.340161+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:11:13.340161+0000","last_undegraded":"2023-10-14T01:11:13.340161+0000","last_fullsized":"2023-10-14T01:11:13.340161+0000","mapping_epoch":15,"log_start":"85'453425","ondisk_log_start":"85'453425","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":310,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 45s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7138697216,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":460673,"num_read_kb":1812160,"num_write":460639,"num_write_kb":1842556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3702,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:11.894983+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:11:11.894983+0000","last_peered":"2023-10-14T01:11:11.894983+0000","last_clean":"2023-10-14T01:11:11.894983+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:11:11.894983+0000","last_undegraded":"2023-10-14T01:11:11.894983+0000","last_fullsized":"2023-10-14T01:11:11.894983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"86'466263","reported_seq":928666,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:12.113645+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:11:12.113645+0000","last_peered":"2023-10-14T01:11:12.113645+0000","last_clean":"2023-10-14T01:11:12.113645+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:11:12.113645+0000","last_undegraded":"2023-10-14T01:11:12.113645+0000","last_fullsized":"2023-10-14T01:11:12.113645+0000","mapping_epoch":15,"log_start":"85'456225","ondisk_log_start":"85'456225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":346,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 52s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7161835520,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":464801,"num_read_kb":1829284,"num_write":463449,"num_write_kb":1853796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1345,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:11.943454+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:11:11.943454+0000","last_peered":"2023-10-14T01:11:11.943454+0000","last_clean":"2023-10-14T01:11:11.943454+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:11:11.943454+0000","last_undegraded":"2023-10-14T01:11:11.943454+0000","last_fullsized":"2023-10-14T01:11:11.943454+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'463413","reported_seq":923762,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:13.339336+0000","last_change":"2023-10-14T01:11:11.094454+0000","last_active":"2023-10-14T01:11:13.339336+0000","last_peered":"2023-10-14T01:11:13.339336+0000","last_clean":"2023-10-14T01:11:13.339336+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:11:13.339336+0000","last_undegraded":"2023-10-14T01:11:13.339336+0000","last_fullsized":"2023-10-14T01:11:13.339336+0000","mapping_epoch":15,"log_start":"85'453333","ondisk_log_start":"85'453333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7143247891,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":462762,"num_read_kb":1820704,"num_write":460841,"num_write_kb":1843358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'12608","reported_seq":17266,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:12.841290+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:11:12.841290+0000","last_peered":"2023-10-14T01:11:12.841290+0000","last_clean":"2023-10-14T01:11:12.841290+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:11:12.841290+0000","last_undegraded":"2023-10-14T01:11:12.841290+0000","last_fullsized":"2023-10-14T01:11:12.841290+0000","mapping_epoch":12,"log_start":"65'2600","ondisk_log_start":"65'2600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":257,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:27.063842+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:10:27.063842+0000","last_peered":"2023-10-14T01:10:27.063842+0000","last_clean":"2023-10-14T01:10:27.063842+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:10:27.063842+0000","last_undegraded":"2023-10-14T01:10:27.063842+0000","last_fullsized":"2023-10-14T01:10:27.063842+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28452110355,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1842093,"num_read_kb":7247836,"num_write":1838155,"num_write_kb":7352614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15058071552,"data_stored":18126098470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739170,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9082028,"kb_used_data":7352420,"kb_used_omap":976,"kb_used_meta":1728623,"kb_avail":85289812,"statfs":{"total":96636764160,"available":87336767488,"internally_reserved":0,"allocated":7528878080,"data_stored":9062403750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1770110237},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:20 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.35999999999999999,"15min":0.39700000000000002},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.41699999999999998,"15min":0.35399999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.492,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.247}]},{"osd":1,"last update":"Sat Oct 14 01:10:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.48799999999999999,"15min":0.50600000000000001},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.17799999999999999},"max":{"1min":4.1740000000000004,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.22},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.53200000000000003,"15min":0.53100000000000003},"min":{"1min":0.27600000000000002,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.309}]}]},{"osd":1,"up_from":8,"seq":34359739171,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5640464,"kb_used_data":3687304,"kb_used_omap":510,"kb_used_meta":1952641,"kb_avail":88731376,"statfs":{"total":96636764160,"available":90860929024,"internally_reserved":0,"allocated":3775799296,"data_stored":4542904869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1999504468},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.47799999999999998,"15min":0.46800000000000003},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.77300000000000002,"5min":4.117,"15min":5.2370000000000001},"last":0.379},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.46600000000000003},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":4.4740000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.27000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.42299999999999999},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.69299999999999995,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.318},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.46800000000000003},"min":{"1min":0.25800000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.74299999999999999,"5min":1.48,"15min":5.8529999999999998},"last":0.34899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739173,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5623416,"kb_used_data":3667632,"kb_used_omap":502,"kb_used_meta":1955273,"kb_avail":88748424,"statfs":{"total":96636764160,"available":90878386176,"internally_reserved":0,"allocated":3755655168,"data_stored":4521211480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":2002200022},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,7,4,18,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.40600000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":1.8109999999999999,"5min":4.452,"15min":8.3239999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.497,"15min":0.41999999999999998},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":3.2709999999999999,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.59099999999999997},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.188},"max":{"1min":4.3390000000000004,"5min":5.875,"15min":26.074999999999999},"last":0.246},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.47699999999999998,"15min":0.41699999999999998},"min":{"1min":0.255,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.78400000000000003,"5min":4.492,"15min":6.5449999999999999},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3754713088,"data_stored":4520898560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3774382080,"data_stored":4542210067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7528976384,"data_stored":9062989843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:18.319 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:18.319 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:18.320 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:18.320 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:18.320 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:38.321 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:38.844 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:11:38.845 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:11:38.880 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2094,"stamp":"2023-10-14T01:11:38.357197+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28487282508,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1873158,"num_read_kb":7301134,"num_write":1847464,"num_write_kb":7388867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86030,"ondisk_log_size":86030,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19539736,"kb_used_data":14778304,"kb_used_omap":1990,"kb_used_meta":4759417,"kb_avail":263575784,"statfs":{"total":289910292480,"available":269901602816,"internally_reserved":0,"allocated":15132983296,"data_stored":18199170851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":4873643847},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,3,7,7,19,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15626240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3897,"num_read_kb":15540,"num_write":3834,"num_write_kb":15336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001687"},"pg_stats":[{"pgid":"2.7","version":"86'9584","reported_seq":13300,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.374438+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:11:32.374438+0000","last_peered":"2023-10-14T01:11:32.374438+0000","last_clean":"2023-10-14T01:11:32.374438+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:11:32.374438+0000","last_undegraded":"2023-10-14T01:11:32.374438+0000","last_fullsized":"2023-10-14T01:11:32.374438+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"86'6456","reported_seq":9885,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.373516+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:11:32.373516+0000","last_peered":"2023-10-14T01:11:32.373516+0000","last_clean":"2023-10-14T01:11:32.373516+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:11:32.373516+0000","last_undegraded":"2023-10-14T01:11:32.373516+0000","last_fullsized":"2023-10-14T01:11:32.373516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":168,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-14T01:10:21.898506+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:10:21.898506+0000","last_peered":"2023-10-14T01:10:21.898506+0000","last_clean":"2023-10-14T01:10:21.898506+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:10:21.898506+0000","last_undegraded":"2023-10-14T01:10:21.898506+0000","last_fullsized":"2023-10-14T01:10:21.898506+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'12740","reported_seq":18064,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.373958+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:11:32.373958+0000","last_peered":"2023-10-14T01:11:32.373958+0000","last_clean":"2023-10-14T01:11:32.373958+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:11:32.373958+0000","last_undegraded":"2023-10-14T01:11:32.373958+0000","last_fullsized":"2023-10-14T01:11:32.373958+0000","mapping_epoch":12,"log_start":"67'2700","ondisk_log_start":"67'2700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'458435","reported_seq":912332,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:37.325335+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:11:37.325335+0000","last_peered":"2023-10-14T01:11:37.325335+0000","last_clean":"2023-10-14T01:11:37.325335+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:11:37.325335+0000","last_undegraded":"2023-10-14T01:11:37.325335+0000","last_fullsized":"2023-10-14T01:11:37.325335+0000","mapping_epoch":15,"log_start":"85'448429","ondisk_log_start":"85'448429","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":547,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 74s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7017672704,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":456229,"num_read_kb":1795168,"num_write":455528,"num_write_kb":1822112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'9445","reported_seq":13049,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:35.796535+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:11:35.796535+0000","last_peered":"2023-10-14T01:11:35.796535+0000","last_clean":"2023-10-14T01:11:35.796535+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:11:35.796535+0000","last_undegraded":"2023-10-14T01:11:35.796535+0000","last_fullsized":"2023-10-14T01:11:35.796535+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9445,"log_dups_size":0,"ondisk_log_size":9445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"86'465360","reported_seq":925525,"reported_epoch":86,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:33.466755+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:11:33.466755+0000","last_peered":"2023-10-14T01:11:33.466755+0000","last_clean":"2023-10-14T01:11:33.466755+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:11:33.466755+0000","last_undegraded":"2023-10-14T01:11:33.466755+0000","last_fullsized":"2023-10-14T01:11:33.466755+0000","mapping_epoch":15,"log_start":"85'455326","ondisk_log_start":"85'455326","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":472,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 65s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7146090496,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":462549,"num_read_kb":1819632,"num_write":462457,"num_write_kb":1849828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3719,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.491417+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:11:37.491417+0000","last_peered":"2023-10-14T01:11:37.491417+0000","last_clean":"2023-10-14T01:11:37.491417+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:11:37.491417+0000","last_undegraded":"2023-10-14T01:11:37.491417+0000","last_fullsized":"2023-10-14T01:11:37.491417+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'468779","reported_seq":933657,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:37.324785+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:11:37.324785+0000","last_peered":"2023-10-14T01:11:37.324785+0000","last_clean":"2023-10-14T01:11:37.324785+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:11:37.324785+0000","last_undegraded":"2023-10-14T01:11:37.324785+0000","last_fullsized":"2023-10-14T01:11:37.324785+0000","mapping_epoch":15,"log_start":"85'458725","ondisk_log_start":"85'458725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":454,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 78s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7171780608,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":467265,"num_read_kb":1839084,"num_write":465891,"num_write_kb":1863564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1351,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.103012+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:11:37.103012+0000","last_peered":"2023-10-14T01:11:37.103012+0000","last_clean":"2023-10-14T01:11:37.103012+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:11:37.103012+0000","last_undegraded":"2023-10-14T01:11:37.103012+0000","last_fullsized":"2023-10-14T01:11:37.103012+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"86'465350","reported_seq":927623,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:33.461965+0000","last_change":"2023-10-14T01:11:31.090272+0000","last_active":"2023-10-14T01:11:33.461965+0000","last_peered":"2023-10-14T01:11:33.461965+0000","last_clean":"2023-10-14T01:11:33.461965+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:11:33.461965+0000","last_undegraded":"2023-10-14T01:11:33.461965+0000","last_fullsized":"2023-10-14T01:11:33.461965+0000","mapping_epoch":15,"log_start":"85'455333","ondisk_log_start":"85'455333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7151144979,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":464678,"num_read_kb":1828336,"num_write":462778,"num_write_kb":1851106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"86'12653","reported_seq":17327,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:32.374856+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:11:32.374856+0000","last_peered":"2023-10-14T01:11:32.374856+0000","last_clean":"2023-10-14T01:11:32.374856+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:11:32.374856+0000","last_undegraded":"2023-10-14T01:11:32.374856+0000","last_fullsized":"2023-10-14T01:11:32.374856+0000","mapping_epoch":12,"log_start":"65'2600","ondisk_log_start":"65'2600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":257,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:10:27.063842+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:10:27.063842+0000","last_peered":"2023-10-14T01:10:27.063842+0000","last_clean":"2023-10-14T01:10:27.063842+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:10:27.063842+0000","last_undegraded":"2023-10-14T01:10:27.063842+0000","last_fullsized":"2023-10-14T01:10:27.063842+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28486688787,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1850721,"num_read_kb":7282220,"num_write":1846654,"num_write_kb":7386610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15131443200,"data_stored":18199470118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40111,"ondisk_log_size":40111,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45820,"ondisk_log_size":45820,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739175,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9248204,"kb_used_data":7389636,"kb_used_omap":976,"kb_used_meta":1857583,"kb_avail":85123636,"statfs":{"total":96636764160,"available":87166603264,"internally_reserved":0,"allocated":7566987264,"data_stored":9100512934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1902165277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.34599999999999997,"15min":0.38800000000000001},"min":{"1min":0.17999999999999999,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.58499999999999996,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.253},{"interface":"front","average":{"1min":0.311,"5min":0.40200000000000002,"15min":0.35899999999999999},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.69899999999999995,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.309}]},{"osd":1,"last update":"Sat Oct 14 01:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.47699999999999998,"15min":0.50700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.72299999999999998,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":0.752,"5min":4.2060000000000004,"15min":16.747},"last":0.47199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739175,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4609016,"kb_used_data":3702192,"kb_used_omap":510,"kb_used_meta":906305,"kb_avail":89762824,"statfs":{"total":96636764160,"available":91917131776,"internally_reserved":0,"allocated":3791044608,"data_stored":4558150181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":928056404},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,1,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.47799999999999998,"15min":0.46800000000000003},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.77300000000000002,"5min":4.117,"15min":5.2370000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.46600000000000003},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":4.4740000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.17799999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.42299999999999999},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.69299999999999995,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.307},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.46800000000000003},"min":{"1min":0.25800000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.74299999999999999,"5min":1.48,"15min":5.8529999999999998},"last":0.34100000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739179,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5682516,"kb_used_data":3686476,"kb_used_omap":502,"kb_used_meta":1995529,"kb_avail":88689324,"statfs":{"total":96636764160,"available":90817867776,"internally_reserved":0,"allocated":3774951424,"data_stored":4540507736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":2043422166},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,7,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.40600000000000003},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":1.8109999999999999,"5min":4.452,"15min":8.3239999999999998},"last":6.1130000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.497,"15min":0.41999999999999998},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":3.2709999999999999,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:10:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.73999999999999999,"15min":0.59099999999999997},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.188},"max":{"1min":4.3390000000000004,"5min":5.875,"15min":26.074999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.47699999999999998,"15min":0.41699999999999998},"min":{"1min":0.255,"5min":0.224,"15min":0.20499999999999999},"max":{"1min":0.78400000000000003,"5min":4.492,"15min":6.5449999999999999},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3774140416,"data_stored":4540325888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3789762560,"data_stored":4557590547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7567540224,"data_stored":9101553683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:38.881 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:38.882 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:38.882 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:38.883 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:38.883 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:11:53.577 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:11:53.578 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:11:53.578 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:11:53.579 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:11:53.579 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:11:53.579 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:11:53.579 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:11:53.580 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:11:53.580 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:11:53.580 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:11:53.580 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:11:53.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:11:53.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:11:53.574+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:11:58.884 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:11:59.432 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:11:59.434 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:11:59.466 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2104,"stamp":"2023-10-14T01:11:58.360502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28517035852,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1880570,"num_read_kb":7330634,"num_write":1854766,"num_write_kb":7418075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86233,"ondisk_log_size":86233,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18408716,"kb_used_data":14838388,"kb_used_omap":1990,"kb_used_meta":3568313,"kb_avail":264706804,"statfs":{"total":289910292480,"available":271059767296,"internally_reserved":0,"allocated":15194509312,"data_stored":18260684438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":3653953351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,11,11,14,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18907136,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4678,"num_read_kb":18628,"num_write":4636,"num_write_kb":18544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001918"},"pg_stats":[{"pgid":"2.7","version":"87'9623","reported_seq":13352,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.579745+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:11:53.579745+0000","last_peered":"2023-10-14T01:11:53.579745+0000","last_clean":"2023-10-14T01:11:53.579745+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:11:53.579745+0000","last_undegraded":"2023-10-14T01:11:53.579745+0000","last_fullsized":"2023-10-14T01:11:53.579745+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6482","reported_seq":9920,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.550579+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:11:53.550579+0000","last_peered":"2023-10-14T01:11:53.550579+0000","last_clean":"2023-10-14T01:11:53.550579+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:11:53.550579+0000","last_undegraded":"2023-10-14T01:11:53.550579+0000","last_fullsized":"2023-10-14T01:11:53.550579+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6482,"log_dups_size":0,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.091983+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:11:37.091983+0000","last_peered":"2023-10-14T01:11:37.091983+0000","last_clean":"2023-10-14T01:11:37.091983+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:11:37.091983+0000","last_undegraded":"2023-10-14T01:11:37.091983+0000","last_fullsized":"2023-10-14T01:11:37.091983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'12792","reported_seq":18134,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.579681+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:11:53.579681+0000","last_peered":"2023-10-14T01:11:53.579681+0000","last_clean":"2023-10-14T01:11:53.579681+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:11:53.579681+0000","last_undegraded":"2023-10-14T01:11:53.579681+0000","last_fullsized":"2023-10-14T01:11:53.579681+0000","mapping_epoch":12,"log_start":"67'2700","ondisk_log_start":"67'2700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'460372","reported_seq":916153,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:57.406350+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:11:57.406350+0000","last_peered":"2023-10-14T01:11:57.406350+0000","last_clean":"2023-10-14T01:11:57.406350+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:11:57.406350+0000","last_undegraded":"2023-10-14T01:11:57.406350+0000","last_fullsized":"2023-10-14T01:11:57.406350+0000","mapping_epoch":15,"log_start":"85'450329","ondisk_log_start":"85'450329","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":703,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 94s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7025098752,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":458094,"num_read_kb":1802584,"num_write":457352,"num_write_kb":1829408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'9493","reported_seq":13110,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:57.037100+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:11:57.037100+0000","last_peered":"2023-10-14T01:11:57.037100+0000","last_clean":"2023-10-14T01:11:57.037100+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:11:57.037100+0000","last_undegraded":"2023-10-14T01:11:57.037100+0000","last_fullsized":"2023-10-14T01:11:57.037100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'467237","reported_seq":929273,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:53.533734+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:11:53.533734+0000","last_peered":"2023-10-14T01:11:53.533734+0000","last_clean":"2023-10-14T01:11:53.533734+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:11:53.533734+0000","last_undegraded":"2023-10-14T01:11:53.533734+0000","last_fullsized":"2023-10-14T01:11:53.533734+0000","mapping_epoch":15,"log_start":"85'457226","ondisk_log_start":"85'457226","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":610,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 85s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7153332224,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":464403,"num_read_kb":1827012,"num_write":464238,"num_write_kb":1856952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3728,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:52.850146+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:11:52.850146+0000","last_peered":"2023-10-14T01:11:52.850146+0000","last_clean":"2023-10-14T01:11:52.850146+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:11:52.850146+0000","last_undegraded":"2023-10-14T01:11:52.850146+0000","last_fullsized":"2023-10-14T01:11:52.850146+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'470704","reported_seq":937384,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:11:57.404117+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:11:57.404117+0000","last_peered":"2023-10-14T01:11:57.404117+0000","last_clean":"2023-10-14T01:11:57.404117+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:11:57.404117+0000","last_undegraded":"2023-10-14T01:11:57.404117+0000","last_fullsized":"2023-10-14T01:11:57.404117+0000","mapping_epoch":15,"log_start":"85'460625","ondisk_log_start":"85'460625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":562,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 98s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7179325440,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":469051,"num_read_kb":1846184,"num_write":467741,"num_write_kb":1870964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1354,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:47.744854+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:11:47.744854+0000","last_peered":"2023-10-14T01:11:47.744854+0000","last_clean":"2023-10-14T01:11:47.744854+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:11:47.744854+0000","last_undegraded":"2023-10-14T01:11:47.744854+0000","last_fullsized":"2023-10-14T01:11:47.744854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'467197","reported_seq":931381,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.579902+0000","last_change":"2023-10-14T01:11:52.188143+0000","last_active":"2023-10-14T01:11:53.579902+0000","last_peered":"2023-10-14T01:11:53.579902+0000","last_clean":"2023-10-14T01:11:53.579902+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:11:53.579902+0000","last_undegraded":"2023-10-14T01:11:53.579902+0000","last_fullsized":"2023-10-14T01:11:53.579902+0000","mapping_epoch":15,"log_start":"85'457133","ondisk_log_start":"85'457133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7158685715,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":466585,"num_read_kb":1835940,"num_write":464625,"num_write_kb":1858494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'12705","reported_seq":17393,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:52.850138+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:11:52.850138+0000","last_peered":"2023-10-14T01:11:52.850138+0000","last_clean":"2023-10-14T01:11:52.850138+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:11:52.850138+0000","last_undegraded":"2023-10-14T01:11:52.850138+0000","last_fullsized":"2023-10-14T01:11:52.850138+0000","mapping_epoch":12,"log_start":"67'2700","ondisk_log_start":"67'2700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:08.294983+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:11:08.294983+0000","last_peered":"2023-10-14T01:11:08.294983+0000","last_clean":"2023-10-14T01:11:08.294983+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:11:08.294983+0000","last_undegraded":"2023-10-14T01:11:08.294983+0000","last_fullsized":"2023-10-14T01:11:08.294983+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28516442131,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1858133,"num_read_kb":7311720,"num_write":1853956,"num_write_kb":7415818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15191453696,"data_stored":18259480614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40197,"ondisk_log_size":40197,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":45937,"ondisk_log_size":45937,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9098296,"kb_used_data":7420272,"kb_used_omap":976,"kb_used_meta":1677039,"kb_avail":85273544,"statfs":{"total":96636764160,"available":87320109056,"internally_reserved":0,"allocated":7598358528,"data_stored":9131880055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1717288221},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.34599999999999997,"15min":0.38800000000000001},"min":{"1min":0.17999999999999999,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.58499999999999996,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.311,"5min":0.40200000000000002,"15min":0.35899999999999999},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.69899999999999995,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.23400000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.47699999999999998,"15min":0.50700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.72299999999999998,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":0.752,"5min":4.2060000000000004,"15min":16.747},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739179,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4677744,"kb_used_data":3716776,"kb_used_omap":510,"kb_used_meta":960449,"kb_avail":89694096,"statfs":{"total":96636764160,"available":91846754304,"internally_reserved":0,"allocated":3805978624,"data_stored":4573080054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":983499860},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,4,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.47799999999999998,"15min":0.46800000000000003},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.77300000000000002,"5min":4.117,"15min":5.2370000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.623,"15min":0.46600000000000003},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.16200000000000001},"max":{"1min":4.4740000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.23300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:10:50 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.42299999999999999},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.69299999999999995,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.38800000000000001,"15min":0.46800000000000003},"min":{"1min":0.25800000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.74299999999999999,"5min":1.48,"15min":5.8529999999999998},"last":0.41999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739183,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4632676,"kb_used_data":3701340,"kb_used_omap":502,"kb_used_meta":930825,"kb_avail":89739164,"statfs":{"total":96636764160,"available":91892903936,"internally_reserved":0,"allocated":3790172160,"data_stored":4555724329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":953165270},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,2,6,2,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.45200000000000001,"15min":0.42599999999999999},"min":{"1min":0.185,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":6.1130000000000004,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48499999999999999,"15min":0.433},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":3.6059999999999999,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.441}]},{"osd":2,"last update":"Sat Oct 14 01:11:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.67700000000000005,"15min":0.51000000000000001},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":2.6139999999999999,"5min":5.875,"15min":5.875},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.45800000000000002,"15min":0.41899999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.20499999999999999},"max":{"1min":0.67200000000000004,"5min":4.492,"15min":6.5449999999999999},"last":0.33200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3789189120,"data_stored":4555374592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3804618752,"data_stored":4572446739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7597645824,"data_stored":9131659283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:11:59.468 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:59.468 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:59.469 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:59.469 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:11:59.470 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:19.472 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:20.005 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:12:20.006 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:12:20.043 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2114,"stamp":"2023-10-14T01:12:18.363997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28547219276,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1888065,"num_read_kb":7360430,"num_write":1862165,"num_write_kb":7447671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86334,"ondisk_log_size":86334,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18631056,"kb_used_data":14897656,"kb_used_omap":1990,"kb_used_meta":3731385,"kb_avail":264484464,"statfs":{"total":289910292480,"available":270832091136,"internally_reserved":0,"allocated":15255199744,"data_stored":18321374870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":3820939079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,2,3,7,19,23,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18579456,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4703,"num_read_kb":18708,"num_write":4552,"num_write_kb":18208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002101"},"pg_stats":[{"pgid":"2.7","version":"87'9671","reported_seq":13412,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:13.330026+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:12:13.330026+0000","last_peered":"2023-10-14T01:12:13.330026+0000","last_clean":"2023-10-14T01:12:13.330026+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:12:13.330026+0000","last_undegraded":"2023-10-14T01:12:13.330026+0000","last_fullsized":"2023-10-14T01:12:13.330026+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9671,"log_dups_size":0,"ondisk_log_size":9671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6514","reported_seq":9961,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:13.329743+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:12:13.329743+0000","last_peered":"2023-10-14T01:12:13.329743+0000","last_clean":"2023-10-14T01:12:13.329743+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:12:13.329743+0000","last_undegraded":"2023-10-14T01:12:13.329743+0000","last_fullsized":"2023-10-14T01:12:13.329743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6514,"log_dups_size":0,"ondisk_log_size":6514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.091983+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:11:37.091983+0000","last_peered":"2023-10-14T01:11:37.091983+0000","last_clean":"2023-10-14T01:11:37.091983+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:11:37.091983+0000","last_undegraded":"2023-10-14T01:11:37.091983+0000","last_fullsized":"2023-10-14T01:11:37.091983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'12856","reported_seq":18214,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:13.329984+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:12:13.329984+0000","last_peered":"2023-10-14T01:12:13.329984+0000","last_clean":"2023-10-14T01:12:13.329984+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:12:13.329984+0000","last_undegraded":"2023-10-14T01:12:13.329984+0000","last_fullsized":"2023-10-14T01:12:13.329984+0000","mapping_epoch":12,"log_start":"67'2800","ondisk_log_start":"67'2800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'462346","reported_seq":919992,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:17.538706+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:12:17.538706+0000","last_peered":"2023-10-14T01:12:17.538706+0000","last_clean":"2023-10-14T01:12:17.538706+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:12:17.538706+0000","last_undegraded":"2023-10-14T01:12:17.538706+0000","last_fullsized":"2023-10-14T01:12:17.538706+0000","mapping_epoch":15,"log_start":"86'452329","ondisk_log_start":"86'452329","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":946,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 114s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7032434688,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":459947,"num_read_kb":1809964,"num_write":459155,"num_write_kb":1836620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'9526","reported_seq":13155,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:16.138443+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:12:16.138443+0000","last_peered":"2023-10-14T01:12:16.138443+0000","last_clean":"2023-10-14T01:12:16.138443+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:12:16.138443+0000","last_undegraded":"2023-10-14T01:12:16.138443+0000","last_fullsized":"2023-10-14T01:12:16.138443+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'469305","reported_seq":933249,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:13.649124+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:12:13.649124+0000","last_peered":"2023-10-14T01:12:13.649124+0000","last_clean":"2023-10-14T01:12:13.649124+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:12:13.649124+0000","last_undegraded":"2023-10-14T01:12:13.649124+0000","last_fullsized":"2023-10-14T01:12:13.649124+0000","mapping_epoch":15,"log_start":"86'459226","ondisk_log_start":"86'459226","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":796,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 105s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7161249792,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":466296,"num_read_kb":1834544,"num_write":466176,"num_write_kb":1864704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3740,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:13.329863+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:12:13.329863+0000","last_peered":"2023-10-14T01:12:13.329863+0000","last_clean":"2023-10-14T01:12:13.329863+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:12:13.329863+0000","last_undegraded":"2023-10-14T01:12:13.329863+0000","last_fullsized":"2023-10-14T01:12:13.329863+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'472680","reported_seq":941238,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:17.567643+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:12:17.567643+0000","last_peered":"2023-10-14T01:12:17.567643+0000","last_clean":"2023-10-14T01:12:17.567643+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:12:17.567643+0000","last_undegraded":"2023-10-14T01:12:17.567643+0000","last_fullsized":"2023-10-14T01:12:17.567643+0000","mapping_epoch":15,"log_start":"86'462625","ondisk_log_start":"86'462625","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":787,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 118s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7186751488,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":470916,"num_read_kb":1853600,"num_write":469557,"num_write_kb":1878228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1358,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:08.209825+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:12:08.209825+0000","last_peered":"2023-10-14T01:12:08.209825+0000","last_clean":"2023-10-14T01:12:08.209825+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:12:08.209825+0000","last_undegraded":"2023-10-14T01:12:08.209825+0000","last_fullsized":"2023-10-14T01:12:08.209825+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'469039","reported_seq":935121,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:13.636750+0000","last_change":"2023-10-14T01:12:12.191033+0000","last_active":"2023-10-14T01:12:13.636750+0000","last_peered":"2023-10-14T01:12:13.636750+0000","last_clean":"2023-10-14T01:12:13.636750+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:12:13.636750+0000","last_undegraded":"2023-10-14T01:12:13.636750+0000","last_fullsized":"2023-10-14T01:12:13.636750+0000","mapping_epoch":15,"log_start":"85'459033","ondisk_log_start":"85'459033","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7166189587,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":468469,"num_read_kb":1843408,"num_write":466467,"num_write_kb":1865862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'12769","reported_seq":17477,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:13.330041+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:12:13.330041+0000","last_peered":"2023-10-14T01:12:13.330041+0000","last_clean":"2023-10-14T01:12:13.330041+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:12:13.330041+0000","last_undegraded":"2023-10-14T01:12:13.330041+0000","last_fullsized":"2023-10-14T01:12:13.330041+0000","mapping_epoch":12,"log_start":"67'2700","ondisk_log_start":"67'2700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:08.294983+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:11:08.294983+0000","last_peered":"2023-10-14T01:11:08.294983+0000","last_clean":"2023-10-14T01:11:08.294983+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:11:08.294983+0000","last_undegraded":"2023-10-14T01:11:08.294983+0000","last_fullsized":"2023-10-14T01:11:08.294983+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28546625555,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1865628,"num_read_kb":7341516,"num_write":1861355,"num_write_kb":7445414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15252082688,"data_stored":18320109606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40157,"ondisk_log_size":40157,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46078,"ondisk_log_size":46078,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739183,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9212552,"kb_used_data":7449920,"kb_used_omap":976,"kb_used_meta":1761647,"kb_avail":85159288,"statfs":{"total":96636764160,"available":87203110912,"internally_reserved":0,"allocated":7628718080,"data_stored":9162239607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1803926813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.34599999999999997,"15min":0.38800000000000001},"min":{"1min":0.17999999999999999,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.58499999999999996,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.311,"5min":0.40200000000000002,"15min":0.35899999999999999},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.69899999999999995,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.20000000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:11:22 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.47699999999999998,"15min":0.50700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.72299999999999998,"5min":4.1840000000000002,"15min":7.8159999999999998},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":0.752,"5min":4.2060000000000004,"15min":16.747},"last":0.34000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739183,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4728904,"kb_used_data":3731840,"kb_used_omap":510,"kb_used_meta":996545,"kb_avail":89642936,"statfs":{"total":96636764160,"available":91794366464,"internally_reserved":0,"allocated":3821404160,"data_stored":4588505590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1020462164},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,9,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46000000000000002,"15min":0.47699999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":1.7929999999999999,"5min":3.0209999999999999,"15min":5.2370000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.51600000000000001,"15min":0.45600000000000002},"min":{"1min":0.17799999999999999,"5min":0.17599999999999999,"15min":0.16200000000000001},"max":{"1min":0.55900000000000005,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.20000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.41699999999999998},"min":{"1min":0.20699999999999999,"5min":0.193,"15min":0.193},"max":{"1min":1.109,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.46200000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":1.1419999999999999,"5min":1.48,"15min":5.8529999999999998},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359739187,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4689600,"kb_used_data":3715896,"kb_used_omap":502,"kb_used_meta":973193,"kb_avail":89682240,"statfs":{"total":96636764160,"available":91834613760,"internally_reserved":0,"allocated":3805077504,"data_stored":4570629673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":996550102},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1,4,10,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.45200000000000001,"15min":0.42599999999999999},"min":{"1min":0.185,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":6.1130000000000004,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48499999999999999,"15min":0.433},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":3.6059999999999999,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:11:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.67700000000000005,"15min":0.51000000000000001},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":2.6139999999999999,"5min":5.875,"15min":5.875},"last":0.252},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.45800000000000002,"15min":0.41899999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.20499999999999999},"max":{"1min":0.67200000000000004,"5min":4.492,"15min":6.5449999999999999},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3804012544,"data_stored":4570198016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3820113920,"data_stored":4587941907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7627956224,"data_stored":9161969683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:12:20.044 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:20.045 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:20.045 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:20.046 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:20.046 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:39.078 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:12:39.078+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:12:39.078 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:12:39.078+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:12:39.502 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:39.502+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:12:39.502 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:39.502+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:12:39.511 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:39.510+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:12:39.511 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:39.510+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:12:39.512 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:39.510+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:12:39.512 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:39.510+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:12:40.047 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:12:40.643 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:12:40.644 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:12:40.692 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2125,"stamp":"2023-10-14T01:12:40.372182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28582661964,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1896840,"num_read_kb":7395318,"num_write":1870880,"num_write_kb":7482531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86409,"ondisk_log_size":86409,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18846500,"kb_used_data":14963276,"kb_used_omap":1990,"kb_used_meta":3881209,"kb_avail":264269020,"statfs":{"total":289910292480,"available":270611476480,"internally_reserved":0,"allocated":15322394624,"data_stored":18388569750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2037945,"internal_metadata":3974358855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,7,8,11,17,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19472384,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4866,"num_read_kb":19364,"num_write":4800,"num_write_kb":19200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001989"},"pg_stats":[{"pgid":"2.7","version":"87'9701","reported_seq":13455,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.527995+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:12:35.527995+0000","last_peered":"2023-10-14T01:12:35.527995+0000","last_clean":"2023-10-14T01:12:35.527995+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:12:35.527995+0000","last_undegraded":"2023-10-14T01:12:35.527995+0000","last_fullsized":"2023-10-14T01:12:35.527995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"87'6534","reported_seq":9989,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.504094+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:12:35.504094+0000","last_peered":"2023-10-14T01:12:35.504094+0000","last_clean":"2023-10-14T01:12:35.504094+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:12:35.504094+0000","last_undegraded":"2023-10-14T01:12:35.504094+0000","last_fullsized":"2023-10-14T01:12:35.504094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6534,"log_dups_size":0,"ondisk_log_size":6534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":170,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.091983+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:11:37.091983+0000","last_peered":"2023-10-14T01:11:37.091983+0000","last_clean":"2023-10-14T01:11:37.091983+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:11:37.091983+0000","last_undegraded":"2023-10-14T01:11:37.091983+0000","last_fullsized":"2023-10-14T01:11:37.091983+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"87'12896","reported_seq":18270,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.534346+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:12:35.534346+0000","last_peered":"2023-10-14T01:12:35.534346+0000","last_clean":"2023-10-14T01:12:35.534346+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:12:35.534346+0000","last_undegraded":"2023-10-14T01:12:35.534346+0000","last_fullsized":"2023-10-14T01:12:35.534346+0000","mapping_epoch":12,"log_start":"67'2800","ondisk_log_start":"67'2800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"87'464342","reported_seq":923907,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:37.652360+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:12:37.652360+0000","last_peered":"2023-10-14T01:12:37.652360+0000","last_clean":"2023-10-14T01:12:37.652360+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:12:37.652360+0000","last_undegraded":"2023-10-14T01:12:37.652360+0000","last_fullsized":"2023-10-14T01:12:37.652360+0000","mapping_epoch":15,"log_start":"86'454329","ondisk_log_start":"86'454329","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1054,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 134s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7040212992,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":461851,"num_read_kb":1817516,"num_write":461075,"num_write_kb":1844300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"87'9566","reported_seq":13207,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:37.422955+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:12:37.422955+0000","last_peered":"2023-10-14T01:12:37.422955+0000","last_clean":"2023-10-14T01:12:37.422955+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:12:37.422955+0000","last_undegraded":"2023-10-14T01:12:37.422955+0000","last_fullsized":"2023-10-14T01:12:37.422955+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9566,"log_dups_size":0,"ondisk_log_size":9566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"87'471820","reported_seq":938153,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:38.753912+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:12:38.753912+0000","last_peered":"2023-10-14T01:12:38.753912+0000","last_clean":"2023-10-14T01:12:38.753912+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:12:38.753912+0000","last_undegraded":"2023-10-14T01:12:38.753912+0000","last_fullsized":"2023-10-14T01:12:38.753912+0000","mapping_epoch":15,"log_start":"86'461726","ondisk_log_start":"86'461726","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":976,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 130s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7170981888,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":468674,"num_read_kb":1844008,"num_write":468569,"num_write_kb":1874276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3752,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:33.815187+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:12:33.815187+0000","last_peered":"2023-10-14T01:12:33.815187+0000","last_clean":"2023-10-14T01:12:33.815187+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:12:33.815187+0000","last_undegraded":"2023-10-14T01:12:33.815187+0000","last_fullsized":"2023-10-14T01:12:33.815187+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"87'474727","reported_seq":945332,"reported_epoch":87,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:12:37.638374+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:12:37.638374+0000","last_peered":"2023-10-14T01:12:37.638374+0000","last_clean":"2023-10-14T01:12:37.638374+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:12:37.638374+0000","last_undegraded":"2023-10-14T01:12:37.638374+0000","last_fullsized":"2023-10-14T01:12:37.638374+0000","mapping_epoch":15,"log_start":"86'464725","ondisk_log_start":"86'464725","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":916,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 138s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7194726400,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":472952,"num_read_kb":1861704,"num_write":471512,"num_write_kb":1886048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1362,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:28.691331+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:12:28.691331+0000","last_peered":"2023-10-14T01:12:28.691331+0000","last_clean":"2023-10-14T01:12:28.691331+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:12:28.691331+0000","last_undegraded":"2023-10-14T01:12:28.691331+0000","last_fullsized":"2023-10-14T01:12:28.691331+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"87'471486","reported_seq":940034,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:38.776468+0000","last_change":"2023-10-14T01:12:36.157448+0000","last_active":"2023-10-14T01:12:38.776468+0000","last_peered":"2023-10-14T01:12:38.776468+0000","last_clean":"2023-10-14T01:12:38.776468+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:12:38.776468+0000","last_undegraded":"2023-10-14T01:12:38.776468+0000","last_fullsized":"2023-10-14T01:12:38.776468+0000","mapping_epoch":15,"log_start":"86'461433","ondisk_log_start":"86'461433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7176146963,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":470926,"num_read_kb":1853176,"num_write":468914,"num_write_kb":1875650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"87'12809","reported_seq":17533,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:12:35.593860+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:12:35.593860+0000","last_peered":"2023-10-14T01:12:35.593860+0000","last_clean":"2023-10-14T01:12:35.593860+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:12:35.593860+0000","last_undegraded":"2023-10-14T01:12:35.593860+0000","last_fullsized":"2023-10-14T01:12:35.593860+0000","mapping_epoch":12,"log_start":"67'2800","ondisk_log_start":"67'2800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":258,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-10-14T01:11:08.294983+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:11:08.294983+0000","last_peered":"2023-10-14T01:11:08.294983+0000","last_clean":"2023-10-14T01:11:08.294983+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:11:08.294983+0000","last_undegraded":"2023-10-14T01:11:08.294983+0000","last_fullsized":"2023-10-14T01:11:08.294983+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28582068243,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1874403,"num_read_kb":7376404,"num_write":1870070,"num_write_kb":7480274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15319461888,"data_stored":18387488806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40162,"ondisk_log_size":40162,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46148,"ondisk_log_size":46148,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739187,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9333688,"kb_used_data":7480880,"kb_used_omap":976,"kb_used_meta":1851823,"kb_avail":85038152,"statfs":{"total":96636764160,"available":87079067648,"internally_reserved":0,"allocated":7660421120,"data_stored":9193942647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000163,"internal_metadata":1896267037},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.38600000000000001},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.63200000000000001,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.376,"15min":0.35799999999999998},"min":{"1min":0.20000000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.32000000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.251}]},{"osd":1,"last update":"Sat Oct 14 01:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.46899999999999997,"15min":0.50900000000000001},"min":{"1min":0.218,"5min":0.189,"15min":0.189},"max":{"1min":1.944,"5min":4.1740000000000004,"15min":7.8159999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51100000000000001,"15min":0.54800000000000004},"min":{"1min":0.26000000000000001,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":1.9770000000000001,"5min":4.2060000000000004,"15min":16.747},"last":0.33800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739188,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4789736,"kb_used_data":3751072,"kb_used_omap":510,"kb_used_meta":1038145,"kb_avail":89582104,"statfs":{"total":96636764160,"available":91732074496,"internally_reserved":0,"allocated":3841097728,"data_stored":4608199158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1063060564},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46000000000000002,"15min":0.47699999999999998},"min":{"1min":0.152,"5min":0.152,"15min":0.152},"max":{"1min":1.7929999999999999,"5min":3.0209999999999999,"15min":5.2370000000000001},"last":0.251},{"interface":"front","average":{"1min":0.26200000000000001,"5min":0.51600000000000001,"15min":0.45600000000000002},"min":{"1min":0.17799999999999999,"5min":0.17599999999999999,"15min":0.16200000000000001},"max":{"1min":0.55900000000000005,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.41699999999999998},"min":{"1min":0.20699999999999999,"5min":0.193,"15min":0.193},"max":{"1min":1.109,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.22},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.46200000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":1.1419999999999999,"5min":1.48,"15min":5.8529999999999998},"last":0.28000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739191,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4723076,"kb_used_data":3731324,"kb_used_omap":502,"kb_used_meta":991241,"kb_avail":89648764,"statfs":{"total":96636764160,"available":91800334336,"internally_reserved":0,"allocated":3820875776,"data_stored":4586427945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1015031254},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:11:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.45200000000000001,"15min":0.42599999999999999},"min":{"1min":0.185,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":6.1130000000000004,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48499999999999999,"15min":0.433},"min":{"1min":0.215,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":3.6059999999999999,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.23100000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:11:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.67700000000000005,"15min":0.51000000000000001},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":2.6139999999999999,"5min":5.875,"15min":5.875},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.45800000000000002,"15min":0.41899999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.20499999999999999},"max":{"1min":0.67200000000000004,"5min":4.492,"15min":6.5449999999999999},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3819884544,"data_stored":4586070016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3839934464,"data_stored":4607762451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7659642880,"data_stored":9193656339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:12:40.694 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:40.694 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:40.695 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:40.695 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:12:40.696 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:12:41.294 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:12:41.322 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:12:41.798 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:12:41.835 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:12:42.328 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:12:42.375 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:12:42.936 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:12:42.985 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:12:44.024 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:44.022+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:12:44.024 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:44.022+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:12:44.040 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:44.038+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:12:44.040 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:44.038+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:12:44.047 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:44.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:12:44.048 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:12:44.046+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:13:02.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:04.410 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:13:04.412 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:13:04.443 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2138,"stamp":"2023-10-14T01:13:04.378248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28621356876,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1906049,"num_read_kb":7431970,"num_write":1880386,"num_write_kb":7520555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86641,"ondisk_log_size":86641,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18194468,"kb_used_data":15039180,"kb_used_omap":1991,"kb_used_meta":3153272,"kb_avail":264921052,"statfs":{"total":289910292480,"available":271279157248,"internally_reserved":0,"allocated":15400120320,"data_stored":18466282888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":3228950711},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,13,14,12,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23449600,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5558,"num_read_kb":22120,"num_write":5765,"num_write_kb":23060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002002"},"pg_stats":[{"pgid":"2.7","version":"88'9764","reported_seq":13534,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:02.300562+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:13:02.300562+0000","last_peered":"2023-10-14T01:13:02.300562+0000","last_clean":"2023-10-14T01:13:02.300562+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:13:02.300562+0000","last_undegraded":"2023-10-14T01:13:02.300562+0000","last_fullsized":"2023-10-14T01:13:02.300562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9764,"log_dups_size":0,"ondisk_log_size":9764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'6576","reported_seq":10042,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:02.280250+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:13:02.280250+0000","last_peered":"2023-10-14T01:13:02.280250+0000","last_clean":"2023-10-14T01:13:02.280250+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:13:02.280250+0000","last_undegraded":"2023-10-14T01:13:02.280250+0000","last_fullsized":"2023-10-14T01:13:02.280250+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":171,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.580672+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:11:53.580672+0000","last_peered":"2023-10-14T01:11:53.580672+0000","last_clean":"2023-10-14T01:11:53.580672+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:11:53.580672+0000","last_undegraded":"2023-10-14T01:11:53.580672+0000","last_fullsized":"2023-10-14T01:11:53.580672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'12978","reported_seq":18373,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:02.313132+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:13:02.313132+0000","last_peered":"2023-10-14T01:13:02.313132+0000","last_clean":"2023-10-14T01:13:02.313132+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:13:02.313132+0000","last_undegraded":"2023-10-14T01:13:02.313132+0000","last_fullsized":"2023-10-14T01:13:02.313132+0000","mapping_epoch":12,"log_start":"69'2900","ondisk_log_start":"69'2900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'466854","reported_seq":928759,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:02.991169+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:13:02.991169+0000","last_peered":"2023-10-14T01:13:02.991169+0000","last_clean":"2023-10-14T01:13:02.991169+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:13:02.991169+0000","last_undegraded":"2023-10-14T01:13:02.991169+0000","last_fullsized":"2023-10-14T01:13:02.991169+0000","mapping_epoch":15,"log_start":"86'456829","ondisk_log_start":"86'456829","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1252,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 159s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7049891840,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":464171,"num_read_kb":1826744,"num_write":463446,"num_write_kb":1853784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"88'9621","reported_seq":13278,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:02.534288+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:13:02.534288+0000","last_peered":"2023-10-14T01:13:02.534288+0000","last_clean":"2023-10-14T01:13:02.534288+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:13:02.534288+0000","last_undegraded":"2023-10-14T01:13:02.534288+0000","last_fullsized":"2023-10-14T01:13:02.534288+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'474306","reported_seq":942975,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:03.917972+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:13:03.917972+0000","last_peered":"2023-10-14T01:13:03.917972+0000","last_clean":"2023-10-14T01:13:03.917972+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:13:03.917972+0000","last_undegraded":"2023-10-14T01:13:03.917972+0000","last_fullsized":"2023-10-14T01:13:03.917972+0000","mapping_epoch":15,"log_start":"86'464226","ondisk_log_start":"86'464226","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1123,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 156s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7180689408,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":470988,"num_read_kb":1853224,"num_write":470954,"num_write_kb":1883816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3768,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:12:59.447829+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:12:59.447829+0000","last_peered":"2023-10-14T01:12:59.447829+0000","last_clean":"2023-10-14T01:12:59.447829+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:12:59.447829+0000","last_undegraded":"2023-10-14T01:12:59.447829+0000","last_fullsized":"2023-10-14T01:12:59.447829+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'477264","reported_seq":950215,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:02.972422+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:13:02.972422+0000","last_peered":"2023-10-14T01:13:02.972422+0000","last_clean":"2023-10-14T01:13:02.972422+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:13:02.972422+0000","last_undegraded":"2023-10-14T01:13:02.972422+0000","last_fullsized":"2023-10-14T01:13:02.972422+0000","mapping_epoch":15,"log_start":"86'467225","ondisk_log_start":"86'467225","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1138,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 163s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7204384768,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":475278,"num_read_kb":1870960,"num_write":473891,"num_write_kb":1895564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1368,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:12:54.311674+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:12:54.311674+0000","last_peered":"2023-10-14T01:12:54.311674+0000","last_clean":"2023-10-14T01:12:54.311674+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:12:54.311674+0000","last_undegraded":"2023-10-14T01:12:54.311674+0000","last_fullsized":"2023-10-14T01:12:54.311674+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'473857","reported_seq":944661,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:03.931068+0000","last_change":"2023-10-14T01:12:57.993681+0000","last_active":"2023-10-14T01:13:03.931068+0000","last_peered":"2023-10-14T01:13:03.931068+0000","last_clean":"2023-10-14T01:13:03.931068+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:13:03.931068+0000","last_undegraded":"2023-10-14T01:13:03.931068+0000","last_fullsized":"2023-10-14T01:13:03.931068+0000","mapping_epoch":15,"log_start":"86'463833","ondisk_log_start":"86'463833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7185797139,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":473175,"num_read_kb":1862128,"num_write":471285,"num_write_kb":1885134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'12893","reported_seq":17638,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:02.325141+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:13:02.325141+0000","last_peered":"2023-10-14T01:13:02.325141+0000","last_clean":"2023-10-14T01:13:02.325141+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:13:02.325141+0000","last_undegraded":"2023-10-14T01:13:02.325141+0000","last_fullsized":"2023-10-14T01:13:02.325141+0000","mapping_epoch":12,"log_start":"67'2800","ondisk_log_start":"67'2800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":259,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:37.105455+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:11:37.105455+0000","last_peered":"2023-10-14T01:11:37.105455+0000","last_clean":"2023-10-14T01:11:37.105455+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:11:37.105455+0000","last_undegraded":"2023-10-14T01:11:37.105455+0000","last_fullsized":"2023-10-14T01:11:37.105455+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28620763155,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1883612,"num_read_kb":7413056,"num_write":1879576,"num_write_kb":7518298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15397019648,"data_stored":18465046566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40168,"ondisk_log_size":40168,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46374,"ondisk_log_size":46374,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739192,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8538580,"kb_used_data":7518540,"kb_used_omap":978,"kb_used_meta":1019053,"kb_avail":85833260,"statfs":{"total":96636764160,"available":87893258240,"internally_reserved":0,"allocated":7698984960,"data_stored":9232502301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1043510925},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,7,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.38600000000000001},"min":{"1min":0.185,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.63200000000000001,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26300000000000001,"5min":0.376,"15min":0.35799999999999998},"min":{"1min":0.20000000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.32000000000000001,"5min":5.1109999999999998,"15min":5.1109999999999998},"last":0.22700000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.46899999999999997,"15min":0.50900000000000001},"min":{"1min":0.218,"5min":0.189,"15min":0.189},"max":{"1min":1.944,"5min":4.1740000000000004,"15min":7.8159999999999998},"last":0.245},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51100000000000001,"15min":0.54800000000000004},"min":{"1min":0.26000000000000001,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":1.9770000000000001,"5min":4.2060000000000004,"15min":16.747},"last":0.30199999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739193,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4848752,"kb_used_data":3769832,"kb_used_omap":510,"kb_used_meta":1078401,"kb_avail":89523088,"statfs":{"total":96636764160,"available":91671642112,"internally_reserved":0,"allocated":3860307968,"data_stored":4627405212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1104282708},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:54 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.41499999999999998,"15min":0.47399999999999998},"min":{"1min":0.17100000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.63,"5min":3.0209999999999999,"15min":5.2370000000000001},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.51600000000000001,"15min":0.46000000000000002},"min":{"1min":0.19800000000000001,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":2.8050000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:12:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36099999999999999,"15min":0.41599999999999998},"min":{"1min":0.22,"5min":0.193,"15min":0.193},"max":{"1min":0.5,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.40500000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.52200000000000002,"5min":1.48,"15min":5.8529999999999998},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359739196,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4807136,"kb_used_data":3750808,"kb_used_omap":502,"kb_used_meta":1055817,"kb_avail":89564704,"statfs":{"total":96636764160,"available":91714256896,"internally_reserved":0,"allocated":3840827392,"data_stored":4606375375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1081157078},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,1,2,4,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.45700000000000002,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":0.57299999999999995,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":3.8959999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.22500000000000001,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":0.59099999999999997,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:12:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.53100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":5.843,"5min":5.875,"15min":5.875},"last":0.25},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.45200000000000001,"15min":0.41799999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":4.492,"15min":6.5449999999999999},"last":3.944}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3839340544,"data_stored":4605526016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3859415040,"data_stored":4627243027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7698264064,"data_stored":9232277523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:13:04.445 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:04.445 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:04.446 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:04.446 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:04.446 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:16.581 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:16.577+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:13:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:16.577+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:13:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:16.577+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:13:16.582 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:16.577+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:13:16.589 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:16.585+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:13:16.590 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:16.585+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:13:24.447 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:24.981 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:13:24.983 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:13:25.029 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2148,"stamp":"2023-10-14T01:13:24.381535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28652924748,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1913499,"num_read_kb":7461606,"num_write":1888150,"num_write_kb":7551611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86553,"ondisk_log_size":86553,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18423240,"kb_used_data":15100848,"kb_used_omap":1991,"kb_used_meta":3320376,"kb_avail":264692280,"statfs":{"total":289910292480,"available":271044894720,"internally_reserved":0,"allocated":15463268352,"data_stored":18529430920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":3400065207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,6,3,13,19,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23576576,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5575,"num_read_kb":22188,"num_write":5797,"num_write_kb":23188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002002"},"pg_stats":[{"pgid":"2.7","version":"88'9794","reported_seq":13576,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:21.834902+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:13:21.834902+0000","last_peered":"2023-10-14T01:13:21.834902+0000","last_clean":"2023-10-14T01:13:21.834902+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:13:21.834902+0000","last_undegraded":"2023-10-14T01:13:21.834902+0000","last_fullsized":"2023-10-14T01:13:21.834902+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'6596","reported_seq":10070,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:21.817432+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:13:21.817432+0000","last_peered":"2023-10-14T01:13:21.817432+0000","last_clean":"2023-10-14T01:13:21.817432+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:13:21.817432+0000","last_undegraded":"2023-10-14T01:13:21.817432+0000","last_fullsized":"2023-10-14T01:13:21.817432+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":171,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.580672+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:11:53.580672+0000","last_peered":"2023-10-14T01:11:53.580672+0000","last_clean":"2023-10-14T01:11:53.580672+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:11:53.580672+0000","last_undegraded":"2023-10-14T01:11:53.580672+0000","last_fullsized":"2023-10-14T01:11:53.580672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'13018","reported_seq":18429,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:21.838068+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:13:21.838068+0000","last_peered":"2023-10-14T01:13:21.838068+0000","last_clean":"2023-10-14T01:13:21.838068+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:13:21.838068+0000","last_undegraded":"2023-10-14T01:13:21.838068+0000","last_fullsized":"2023-10-14T01:13:21.838068+0000","mapping_epoch":12,"log_start":"69'3000","ondisk_log_start":"69'3000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'468905","reported_seq":932645,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:23.076658+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:13:23.076658+0000","last_peered":"2023-10-14T01:13:23.076658+0000","last_clean":"2023-10-14T01:13:23.076658+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:13:23.076658+0000","last_undegraded":"2023-10-14T01:13:23.076658+0000","last_fullsized":"2023-10-14T01:13:23.076658+0000","mapping_epoch":15,"log_start":"87'458829","ondisk_log_start":"87'458829","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1417,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 179s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7057723392,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":465993,"num_read_kb":1833980,"num_write":465379,"num_write_kb":1861516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"88'9651","reported_seq":13320,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:22.969218+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:13:22.969218+0000","last_peered":"2023-10-14T01:13:22.969218+0000","last_clean":"2023-10-14T01:13:22.969218+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:13:22.969218+0000","last_undegraded":"2023-10-14T01:13:22.969218+0000","last_fullsized":"2023-10-14T01:13:22.969218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'476331","reported_seq":946911,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:24.000753+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:13:24.000753+0000","last_peered":"2023-10-14T01:13:24.000753+0000","last_clean":"2023-10-14T01:13:24.000753+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:13:24.000753+0000","last_undegraded":"2023-10-14T01:13:24.000753+0000","last_fullsized":"2023-10-14T01:13:24.000753+0000","mapping_epoch":15,"log_start":"87'466326","ondisk_log_start":"87'466326","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1228,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 176s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7188611072,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":472890,"num_read_kb":1860788,"num_write":472904,"num_write_kb":1891616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3780,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:19.943533+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:13:19.943533+0000","last_peered":"2023-10-14T01:13:19.943533+0000","last_clean":"2023-10-14T01:13:19.943533+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:13:19.943533+0000","last_undegraded":"2023-10-14T01:13:19.943533+0000","last_fullsized":"2023-10-14T01:13:19.943533+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'479356","reported_seq":954115,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:23.078885+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:13:23.078885+0000","last_peered":"2023-10-14T01:13:23.078885+0000","last_clean":"2023-10-14T01:13:23.078885+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:13:23.078885+0000","last_undegraded":"2023-10-14T01:13:23.078885+0000","last_fullsized":"2023-10-14T01:13:23.078885+0000","mapping_epoch":15,"log_start":"87'469326","ondisk_log_start":"87'469326","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1276,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 183s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7212490752,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":477075,"num_read_kb":1878124,"num_write":475887,"num_write_kb":1903548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1373,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:14.773672+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:13:14.773672+0000","last_peered":"2023-10-14T01:13:14.773672+0000","last_clean":"2023-10-14T01:13:14.773672+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:13:14.773672+0000","last_undegraded":"2023-10-14T01:13:14.773672+0000","last_fullsized":"2023-10-14T01:13:14.773672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'475742","reported_seq":948481,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:23.996949+0000","last_change":"2023-10-14T01:13:23.070774+0000","last_active":"2023-10-14T01:13:23.996949+0000","last_peered":"2023-10-14T01:13:23.996949+0000","last_clean":"2023-10-14T01:13:23.996949+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:13:23.996949+0000","last_undegraded":"2023-10-14T01:13:23.996949+0000","last_fullsized":"2023-10-14T01:13:23.996949+0000","mapping_epoch":15,"log_start":"87'465733","ondisk_log_start":"87'465733","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7193505811,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":475104,"num_read_kb":1869800,"num_write":473170,"num_write_kb":1892674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'12933","reported_seq":17694,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:21.907272+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:13:21.907272+0000","last_peered":"2023-10-14T01:13:21.907272+0000","last_clean":"2023-10-14T01:13:21.907272+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:13:21.907272+0000","last_undegraded":"2023-10-14T01:13:21.907272+0000","last_fullsized":"2023-10-14T01:13:21.907272+0000","mapping_epoch":12,"log_start":"69'2900","ondisk_log_start":"69'2900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:12:47.120715+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:12:47.120715+0000","last_peered":"2023-10-14T01:12:47.120715+0000","last_clean":"2023-10-14T01:12:47.120715+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:12:47.120715+0000","last_undegraded":"2023-10-14T01:12:47.120715+0000","last_fullsized":"2023-10-14T01:12:47.120715+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28652331027,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1891062,"num_read_kb":7442692,"num_write":1887340,"num_write_kb":7549354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15460610048,"data_stored":18528636966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40120,"ondisk_log_size":40120,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46334,"ondisk_log_size":46334,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739196,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8635840,"kb_used_data":7549240,"kb_used_omap":978,"kb_used_meta":1085613,"kb_avail":85736000,"statfs":{"total":96636764160,"available":87793664000,"internally_reserved":0,"allocated":7730421760,"data_stored":9263939101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1111668365},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.38800000000000001},"min":{"1min":0.17000000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.93600000000000005,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.32800000000000001,"15min":0.35699999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.33800000000000002,"5min":3.4950000000000001,"15min":5.1109999999999998},"last":0.245}]},{"osd":1,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.45600000000000002,"15min":0.48399999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52100000000000002,"5min":4.1740000000000004,"15min":7.8159999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.501,"15min":0.55100000000000005},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.54800000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739197,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4924308,"kb_used_data":3785100,"kb_used_omap":510,"kb_used_meta":1138689,"kb_avail":89447532,"statfs":{"total":96636764160,"available":91594272768,"internally_reserved":0,"allocated":3875942400,"data_stored":4643039644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1166017620},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:54 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.41499999999999998,"15min":0.47399999999999998},"min":{"1min":0.17100000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.63,"5min":3.0209999999999999,"15min":5.2370000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.51600000000000001,"15min":0.46000000000000002},"min":{"1min":0.19800000000000001,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":2.8050000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":0.5}]},{"osd":2,"last update":"Sat Oct 14 01:12:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36099999999999999,"15min":0.41599999999999998},"min":{"1min":0.22,"5min":0.193,"15min":0.193},"max":{"1min":0.5,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.40500000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.52200000000000002,"5min":1.48,"15min":5.8529999999999998},"last":0.54900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739200,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4863092,"kb_used_data":3766508,"kb_used_omap":502,"kb_used_meta":1096073,"kb_avail":89508748,"statfs":{"total":96636764160,"available":91656957952,"internally_reserved":0,"allocated":3856904192,"data_stored":4622452175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1122379222},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,1,7,16,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.45700000000000002,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":0.57299999999999995,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.248},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.22500000000000001,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":0.59099999999999997,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.30399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:12:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.53100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":5.843,"5min":5.875,"15min":5.875},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.45200000000000001,"15min":0.41799999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":4.492,"15min":6.5449999999999999},"last":0.28299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3855433728,"data_stored":4621619200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3875123200,"data_stored":4642951187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7730053120,"data_stored":9264066579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:13:25.031 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:25.031 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:25.031 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:25.032 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:25.032 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:13:35.699 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:35.697+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:13:35.700 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:35.697+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:13:35.707 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:35.705+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:13:35.707 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:35.705+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:13:35.716 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:35.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:13:35.717 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:13:35.713+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:13:45.033 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:13:45.619 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:13:45.621 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:13:45.658 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2158,"stamp":"2023-10-14T01:13:44.385258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28684042060,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1920676,"num_read_kb":7490222,"num_write":1895807,"num_write_kb":7582239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86796,"ondisk_log_size":86796,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18636476,"kb_used_data":15162084,"kb_used_omap":1991,"kb_used_meta":3472376,"kb_avail":264479044,"statfs":{"total":289910292480,"available":270826541056,"internally_reserved":0,"allocated":15525974016,"data_stored":18592136584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":3555713207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,8,25,20],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23453696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5449,"num_read_kb":21732,"num_write":5774,"num_write_kb":23096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002015"},"pg_stats":[{"pgid":"2.7","version":"88'9860","reported_seq":13654,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.761064+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:13:41.761064+0000","last_peered":"2023-10-14T01:13:41.761064+0000","last_clean":"2023-10-14T01:13:41.761064+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:13:41.761064+0000","last_undegraded":"2023-10-14T01:13:41.761064+0000","last_fullsized":"2023-10-14T01:13:41.761064+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9860,"log_dups_size":0,"ondisk_log_size":9860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'6640","reported_seq":10122,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.747593+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:13:41.747593+0000","last_peered":"2023-10-14T01:13:41.747593+0000","last_clean":"2023-10-14T01:13:41.747593+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:13:41.747593+0000","last_undegraded":"2023-10-14T01:13:41.747593+0000","last_fullsized":"2023-10-14T01:13:41.747593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6640,"log_dups_size":0,"ondisk_log_size":6640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":171,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-14T01:11:53.580672+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:11:53.580672+0000","last_peered":"2023-10-14T01:11:53.580672+0000","last_clean":"2023-10-14T01:11:53.580672+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:11:53.580672+0000","last_undegraded":"2023-10-14T01:11:53.580672+0000","last_fullsized":"2023-10-14T01:11:53.580672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"88'13106","reported_seq":18533,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.777306+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:13:41.777306+0000","last_peered":"2023-10-14T01:13:41.777306+0000","last_clean":"2023-10-14T01:13:41.777306+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:13:41.777306+0000","last_undegraded":"2023-10-14T01:13:41.777306+0000","last_fullsized":"2023-10-14T01:13:41.777306+0000","mapping_epoch":12,"log_start":"69'3100","ondisk_log_start":"69'3100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"88'470869","reported_seq":936390,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:43.224024+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:13:43.224024+0000","last_peered":"2023-10-14T01:13:43.224024+0000","last_clean":"2023-10-14T01:13:43.224024+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:13:43.224024+0000","last_undegraded":"2023-10-14T01:13:43.224024+0000","last_fullsized":"2023-10-14T01:13:43.224024+0000","mapping_epoch":15,"log_start":"87'460829","ondisk_log_start":"87'460829","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1576,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 200s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7065272320,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":467761,"num_read_kb":1841032,"num_write":467233,"num_write_kb":1868932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"88'9696","reported_seq":13377,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:42.289046+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:13:42.289046+0000","last_peered":"2023-10-14T01:13:42.289046+0000","last_clean":"2023-10-14T01:13:42.289046+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:13:42.289046+0000","last_undegraded":"2023-10-14T01:13:42.289046+0000","last_fullsized":"2023-10-14T01:13:42.289046+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9696,"log_dups_size":0,"ondisk_log_size":9696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"88'478311","reported_seq":950719,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:44.100603+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:13:44.100603+0000","last_peered":"2023-10-14T01:13:44.100603+0000","last_clean":"2023-10-14T01:13:44.100603+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:13:44.100603+0000","last_undegraded":"2023-10-14T01:13:44.100603+0000","last_fullsized":"2023-10-14T01:13:44.100603+0000","mapping_epoch":15,"log_start":"87'468226","ondisk_log_start":"87'468226","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1327,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 196s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7196356608,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":474706,"num_read_kb":1868020,"num_write":474816,"num_write_kb":1899264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3793,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:40.370005+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:13:40.370005+0000","last_peered":"2023-10-14T01:13:40.370005+0000","last_clean":"2023-10-14T01:13:40.370005+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:13:40.370005+0000","last_undegraded":"2023-10-14T01:13:40.370005+0000","last_fullsized":"2023-10-14T01:13:40.370005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"88'481409","reported_seq":957972,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:13:43.198180+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:13:43.198180+0000","last_peered":"2023-10-14T01:13:43.198180+0000","last_clean":"2023-10-14T01:13:43.198180+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:13:43.198180+0000","last_undegraded":"2023-10-14T01:13:43.198180+0000","last_fullsized":"2023-10-14T01:13:43.198180+0000","mapping_epoch":15,"log_start":"87'471326","ondisk_log_start":"87'471326","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1384,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 204s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7220494336,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":478866,"num_read_kb":1885284,"num_write":477863,"num_write_kb":1911452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1377,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:35.261707+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:13:35.261707+0000","last_peered":"2023-10-14T01:13:35.261707+0000","last_clean":"2023-10-14T01:13:35.261707+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:13:35.261707+0000","last_undegraded":"2023-10-14T01:13:35.261707+0000","last_fullsized":"2023-10-14T01:13:35.261707+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"88'477657","reported_seq":952203,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:44.094091+0000","last_change":"2023-10-14T01:13:42.118872+0000","last_active":"2023-10-14T01:13:44.094091+0000","last_peered":"2023-10-14T01:13:44.094091+0000","last_clean":"2023-10-14T01:13:44.094091+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:13:44.094091+0000","last_undegraded":"2023-10-14T01:13:44.094091+0000","last_fullsized":"2023-10-14T01:13:44.094091+0000","mapping_epoch":15,"log_start":"87'467633","ondisk_log_start":"87'467633","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7201325075,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":476906,"num_read_kb":1876972,"num_write":475085,"num_write_kb":1900334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'13021","reported_seq":17798,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:41.801650+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:13:41.801650+0000","last_peered":"2023-10-14T01:13:41.801650+0000","last_clean":"2023-10-14T01:13:41.801650+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:13:41.801650+0000","last_undegraded":"2023-10-14T01:13:41.801650+0000","last_fullsized":"2023-10-14T01:13:41.801650+0000","mapping_epoch":12,"log_start":"69'3000","ondisk_log_start":"69'3000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":260,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:12:47.120715+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:12:47.120715+0000","last_peered":"2023-10-14T01:12:47.120715+0000","last_clean":"2023-10-14T01:12:47.120715+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:12:47.120715+0000","last_undegraded":"2023-10-14T01:12:47.120715+0000","last_fullsized":"2023-10-14T01:12:47.120715+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28683448339,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1898239,"num_read_kb":7471308,"num_write":1894997,"num_write_kb":7579982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15523278848,"data_stored":18591305766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46465,"ondisk_log_size":46465,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8756700,"kb_used_data":7579924,"kb_used_omap":978,"kb_used_meta":1175789,"kb_avail":85615140,"statfs":{"total":96636764160,"available":87669903360,"internally_reserved":0,"allocated":7761842176,"data_stored":9295359517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1204008589},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.38800000000000001},"min":{"1min":0.17000000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.93600000000000005,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.192},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.32800000000000001,"15min":0.35699999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.33800000000000002,"5min":3.4950000000000001,"15min":5.1109999999999998},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.45600000000000002,"15min":0.48399999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52100000000000002,"5min":4.1740000000000004,"15min":7.8159999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.501,"15min":0.55100000000000005},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739201,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4975688,"kb_used_data":3800384,"kb_used_omap":510,"kb_used_meta":1174785,"kb_avail":89396152,"statfs":{"total":96636764160,"available":91541659648,"internally_reserved":0,"allocated":3891593216,"data_stored":4658690460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1202979924},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,12,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:12:54 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.41499999999999998,"15min":0.47399999999999998},"min":{"1min":0.17100000000000001,"5min":0.152,"15min":0.152},"max":{"1min":0.63,"5min":3.0209999999999999,"15min":5.2370000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.51600000000000001,"15min":0.46000000000000002},"min":{"1min":0.19800000000000001,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":2.8050000000000002,"5min":8.8089999999999993,"15min":8.8089999999999993},"last":2.593}]},{"osd":2,"last update":"Sat Oct 14 01:12:54 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36099999999999999,"15min":0.41599999999999998},"min":{"1min":0.22,"5min":0.193,"15min":0.193},"max":{"1min":0.5,"5min":1.4279999999999999,"15min":5.6040000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.40500000000000003,"15min":0.46200000000000002},"min":{"1min":0.28000000000000003,"5min":0.25600000000000001,"15min":0.223},"max":{"1min":0.52200000000000002,"5min":1.48,"15min":5.8529999999999998},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739204,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904088,"kb_used_data":3781776,"kb_used_omap":502,"kb_used_meta":1121801,"kb_avail":89467752,"statfs":{"total":96636764160,"available":91614978048,"internally_reserved":0,"allocated":3872538624,"data_stored":4638086607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1148724694},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,10,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:12:43 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.45700000000000002,"15min":0.42399999999999999},"min":{"1min":0.24299999999999999,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":0.57299999999999995,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.252},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.22500000000000001,"5min":0.17399999999999999,"15min":0.16400000000000001},"max":{"1min":0.59099999999999997,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.38900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:12:43 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.53100000000000003},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.188},"max":{"1min":5.843,"5min":5.875,"15min":5.875},"last":3.3330000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.45200000000000001,"15min":0.41799999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":4.492,"15min":6.5449999999999999},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3871129600,"data_stored":4637315072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3890794496,"data_stored":4658622483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7761354752,"data_stored":9295368211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:13:45.660 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:45.660 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:45.661 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:45.661 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:13:45.662 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:05.663 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:06.205 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:14:06.206 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:14:06.240 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2169,"stamp":"2023-10-14T01:14:04.388530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28715495244,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1927990,"num_read_kb":7519398,"num_write":1903538,"num_write_kb":7613163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86893,"ondisk_log_size":86893,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18919152,"kb_used_data":15224216,"kb_used_omap":1991,"kb_used_meta":3692920,"kb_avail":264196368,"statfs":{"total":289910292480,"available":270537080832,"internally_reserved":0,"allocated":15589597184,"data_stored":18655747065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":3781550263},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,3,8,9,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23744512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5552,"num_read_kb":22148,"num_write":5842,"num_write_kb":23368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001877"},"pg_stats":[{"pgid":"2.7","version":"89'9890","reported_seq":13697,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:03.553157+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:14:03.553157+0000","last_peered":"2023-10-14T01:14:03.553157+0000","last_clean":"2023-10-14T01:14:03.553157+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:14:03.553157+0000","last_undegraded":"2023-10-14T01:14:03.553157+0000","last_fullsized":"2023-10-14T01:14:03.553157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6660","reported_seq":10151,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:03.538062+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:14:03.538062+0000","last_peered":"2023-10-14T01:14:03.538062+0000","last_clean":"2023-10-14T01:14:03.538062+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:14:03.538062+0000","last_undegraded":"2023-10-14T01:14:03.538062+0000","last_fullsized":"2023-10-14T01:14:03.538062+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":173,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:52.124812+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:13:52.124812+0000","last_peered":"2023-10-14T01:13:52.124812+0000","last_clean":"2023-10-14T01:13:52.124812+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:13:52.124812+0000","last_undegraded":"2023-10-14T01:13:52.124812+0000","last_fullsized":"2023-10-14T01:13:52.124812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'13146","reported_seq":18591,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:03.569387+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:14:03.569387+0000","last_peered":"2023-10-14T01:14:03.569387+0000","last_clean":"2023-10-14T01:14:03.569387+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:14:03.569387+0000","last_undegraded":"2023-10-14T01:14:03.569387+0000","last_fullsized":"2023-10-14T01:14:03.569387+0000","mapping_epoch":12,"log_start":"69'3100","ondisk_log_start":"69'3100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'472887","reported_seq":940144,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:03.284265+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:14:03.284265+0000","last_peered":"2023-10-14T01:14:03.284265+0000","last_clean":"2023-10-14T01:14:03.284265+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:14:03.284265+0000","last_undegraded":"2023-10-14T01:14:03.284265+0000","last_fullsized":"2023-10-14T01:14:03.284265+0000","mapping_epoch":15,"log_start":"87'462829","ondisk_log_start":"87'462829","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1732,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 220s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7072985088,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":469482,"num_read_kb":1847892,"num_write":469140,"num_write_kb":1876560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'9732","reported_seq":13427,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:02.078277+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:14:02.078277+0000","last_peered":"2023-10-14T01:14:02.078277+0000","last_clean":"2023-10-14T01:14:02.078277+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:14:02.078277+0000","last_undegraded":"2023-10-14T01:14:02.078277+0000","last_fullsized":"2023-10-14T01:14:02.078277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'480417","reported_seq":954662,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:04.234079+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:14:04.234079+0000","last_peered":"2023-10-14T01:14:04.234079+0000","last_clean":"2023-10-14T01:14:04.234079+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:14:04.234079+0000","last_undegraded":"2023-10-14T01:14:04.234079+0000","last_fullsized":"2023-10-14T01:14:04.234079+0000","mapping_epoch":15,"log_start":"87'470326","ondisk_log_start":"87'470326","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1459,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 216s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7204544512,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":476527,"num_read_kb":1875292,"num_write":476828,"num_write_kb":1907312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3806,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:00.856078+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:14:00.856078+0000","last_peered":"2023-10-14T01:14:00.856078+0000","last_clean":"2023-10-14T01:14:00.856078+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:14:00.856078+0000","last_undegraded":"2023-10-14T01:14:00.856078+0000","last_fullsized":"2023-10-14T01:14:00.856078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'483433","reported_seq":961900,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:03.290371+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:14:03.290371+0000","last_peered":"2023-10-14T01:14:03.290371+0000","last_clean":"2023-10-14T01:14:03.290371+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:14:03.290371+0000","last_undegraded":"2023-10-14T01:14:03.290371+0000","last_fullsized":"2023-10-14T01:14:03.290371+0000","mapping_epoch":15,"log_start":"87'473426","ondisk_log_start":"87'473426","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1522,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 224s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7228383232,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":480756,"num_read_kb":1892820,"num_write":479792,"num_write_kb":1919168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1382,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:55.729592+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:13:55.729592+0000","last_peered":"2023-10-14T01:13:55.729592+0000","last_clean":"2023-10-14T01:13:55.729592+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:13:55.729592+0000","last_undegraded":"2023-10-14T01:13:55.729592+0000","last_fullsized":"2023-10-14T01:13:55.729592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'479540","reported_seq":955981,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:04.252204+0000","last_change":"2023-10-14T01:14:03.392320+0000","last_active":"2023-10-14T01:14:04.252204+0000","last_peered":"2023-10-14T01:14:04.252204+0000","last_clean":"2023-10-14T01:14:04.252204+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:14:04.252204+0000","last_undegraded":"2023-10-14T01:14:04.252204+0000","last_fullsized":"2023-10-14T01:14:04.252204+0000","mapping_epoch":15,"log_start":"87'469533","ondisk_log_start":"87'469533","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7208988691,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":478788,"num_read_kb":1884480,"num_write":476968,"num_write_kb":1907866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'13061","reported_seq":17856,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:03.554698+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:14:03.554698+0000","last_peered":"2023-10-14T01:14:03.554698+0000","last_clean":"2023-10-14T01:14:03.554698+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:14:03.554698+0000","last_undegraded":"2023-10-14T01:14:03.554698+0000","last_fullsized":"2023-10-14T01:14:03.554698+0000","mapping_epoch":12,"log_start":"69'3000","ondisk_log_start":"69'3000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:08.956610+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:13:08.956610+0000","last_peered":"2023-10-14T01:13:08.956610+0000","last_clean":"2023-10-14T01:13:08.956610+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:13:08.956610+0000","last_undegraded":"2023-10-14T01:13:08.956610+0000","last_fullsized":"2023-10-14T01:13:08.956610+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28714901523,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1905553,"num_read_kb":7500484,"num_write":1902728,"num_write_kb":7610906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15586607104,"data_stored":18654634022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40163,"ondisk_log_size":40163,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46631,"ondisk_log_size":46631,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8911468,"kb_used_data":7610980,"kb_used_omap":978,"kb_used_meta":1299501,"kb_avail":85460372,"statfs":{"total":96636764160,"available":87511420928,"internally_reserved":0,"allocated":7793643520,"data_stored":9327156632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1330689677},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.38800000000000001},"min":{"1min":0.17000000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.93600000000000005,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.221},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.32800000000000001,"15min":0.35699999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.33800000000000002,"5min":3.4950000000000001,"15min":5.1109999999999998},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.45600000000000002,"15min":0.48399999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52100000000000002,"5min":4.1740000000000004,"15min":7.8159999999999998},"last":0.317},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.501,"15min":0.55100000000000005},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.35399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739205,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5052060,"kb_used_data":3816020,"kb_used_omap":510,"kb_used_meta":1235521,"kb_avail":89319780,"statfs":{"total":96636764160,"available":91463454720,"internally_reserved":0,"allocated":3907604480,"data_stored":4674697495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1265173588},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.442,"15min":0.47499999999999998},"min":{"1min":0.18099999999999999,"5min":0.152,"15min":0.152},"max":{"1min":3.7130000000000001,"5min":3.7130000000000001,"15min":5.2370000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.45600000000000002,"15min":0.46899999999999997},"min":{"1min":0.21199999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":8.8089999999999993},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.20999999999999999,"5min":0.193,"15min":0.193},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.442,"15min":0.46100000000000002},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.255},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.36899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739208,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4955624,"kb_used_data":3797216,"kb_used_omap":502,"kb_used_meta":1157897,"kb_avail":89416216,"statfs":{"total":96636764160,"available":91562205184,"internally_reserved":0,"allocated":3888349184,"data_stored":4653892938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1185686998},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,6,14,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.436},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":3.8959999999999999,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.45200000000000001,"15min":0.434},"min":{"1min":0.191,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":0.77100000000000002,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.59099999999999997,"15min":0.53400000000000003},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.188},"max":{"1min":3.3330000000000002,"5min":5.843,"15min":5.875},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.41099999999999998,"15min":0.43099999999999999},"min":{"1min":0.27500000000000002,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":3.944,"5min":4.492,"15min":6.5449999999999999},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3886841856,"data_stored":4653027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3906748416,"data_stored":4674576403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7793016832,"data_stored":9327030291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:14:06.242 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:06.242 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:06.243 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:06.243 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:06.243 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:10.790 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:14:10.788+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:14:10.791 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:14:10.788+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:14:15.722 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:14:15.720+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:14:26.244 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:26.807 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:14:26.809 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:14:26.837 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2180,"stamp":"2023-10-14T01:14:26.392223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28746870604,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1935322,"num_read_kb":7548666,"num_write":1911265,"num_write_kb":7644071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86906,"ondisk_log_size":86906,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19161864,"kb_used_data":15286512,"kb_used_omap":1991,"kb_used_meta":3873336,"kb_avail":263953656,"statfs":{"total":289910292480,"available":270288543744,"internally_reserved":0,"allocated":15653388288,"data_stored":18719538169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":3966296247},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,7,5,14,14,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15642624,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3661,"num_read_kb":14612,"num_write":3862,"num_write_kb":15448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002105"},"pg_stats":[{"pgid":"2.7","version":"89'9920","reported_seq":13739,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:24.128593+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:14:24.128593+0000","last_peered":"2023-10-14T01:14:24.128593+0000","last_clean":"2023-10-14T01:14:24.128593+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:14:24.128593+0000","last_undegraded":"2023-10-14T01:14:24.128593+0000","last_fullsized":"2023-10-14T01:14:24.128593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9920,"log_dups_size":0,"ondisk_log_size":9920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6681","reported_seq":10181,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:24.109391+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:14:24.109391+0000","last_peered":"2023-10-14T01:14:24.109391+0000","last_clean":"2023-10-14T01:14:24.109391+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:14:24.109391+0000","last_undegraded":"2023-10-14T01:14:24.109391+0000","last_fullsized":"2023-10-14T01:14:24.109391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6681,"log_dups_size":0,"ondisk_log_size":6681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":173,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:52.124812+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:13:52.124812+0000","last_peered":"2023-10-14T01:13:52.124812+0000","last_clean":"2023-10-14T01:13:52.124812+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:13:52.124812+0000","last_undegraded":"2023-10-14T01:13:52.124812+0000","last_fullsized":"2023-10-14T01:13:52.124812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'13186","reported_seq":18647,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:24.128853+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:14:24.128853+0000","last_peered":"2023-10-14T01:14:24.128853+0000","last_clean":"2023-10-14T01:14:24.128853+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:14:24.128853+0000","last_undegraded":"2023-10-14T01:14:24.128853+0000","last_fullsized":"2023-10-14T01:14:24.128853+0000","mapping_epoch":12,"log_start":"69'3100","ondisk_log_start":"69'3100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'474935","reported_seq":943999,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:23.431199+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:14:23.431199+0000","last_peered":"2023-10-14T01:14:23.431199+0000","last_clean":"2023-10-14T01:14:23.431199+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:14:23.431199+0000","last_undegraded":"2023-10-14T01:14:23.431199+0000","last_fullsized":"2023-10-14T01:14:23.431199+0000","mapping_epoch":15,"log_start":"87'464930","ondisk_log_start":"87'464930","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1876,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 240s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7080927232,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":471274,"num_read_kb":1855052,"num_write":471089,"num_write_kb":1884356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'9792","reported_seq":13499,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:22.649746+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:14:22.649746+0000","last_peered":"2023-10-14T01:14:22.649746+0000","last_clean":"2023-10-14T01:14:22.649746+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:14:22.649746+0000","last_undegraded":"2023-10-14T01:14:22.649746+0000","last_fullsized":"2023-10-14T01:14:22.649746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'482428","reported_seq":958516,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:24.389112+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:14:24.389112+0000","last_peered":"2023-10-14T01:14:24.389112+0000","last_clean":"2023-10-14T01:14:24.389112+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:14:24.389112+0000","last_undegraded":"2023-10-14T01:14:24.389112+0000","last_fullsized":"2023-10-14T01:14:24.389112+0000","mapping_epoch":15,"log_start":"87'472426","ondisk_log_start":"87'472426","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1594,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 236s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7212302336,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":478355,"num_read_kb":1882584,"num_write":478743,"num_write_kb":1914972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3818,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:21.331601+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:14:21.331601+0000","last_peered":"2023-10-14T01:14:21.331601+0000","last_clean":"2023-10-14T01:14:21.331601+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:14:21.331601+0000","last_undegraded":"2023-10-14T01:14:21.331601+0000","last_fullsized":"2023-10-14T01:14:21.331601+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'485464","reported_seq":965815,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:23.424067+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:14:23.424067+0000","last_peered":"2023-10-14T01:14:23.424067+0000","last_clean":"2023-10-14T01:14:23.424067+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:14:23.424067+0000","last_undegraded":"2023-10-14T01:14:23.424067+0000","last_fullsized":"2023-10-14T01:14:23.424067+0000","mapping_epoch":15,"log_start":"87'475426","ondisk_log_start":"87'475426","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1666,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 244s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7236210688,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":482624,"num_read_kb":1900264,"num_write":481722,"num_write_kb":1926888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1386,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:16.211718+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:14:16.211718+0000","last_peered":"2023-10-14T01:14:16.211718+0000","last_clean":"2023-10-14T01:14:16.211718+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:14:16.211718+0000","last_undegraded":"2023-10-14T01:14:16.211718+0000","last_fullsized":"2023-10-14T01:14:16.211718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'481473","reported_seq":959763,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:24.406282+0000","last_change":"2023-10-14T01:14:21.393023+0000","last_active":"2023-10-14T01:14:24.406282+0000","last_peered":"2023-10-14T01:14:24.406282+0000","last_clean":"2023-10-14T01:14:24.406282+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:14:24.406282+0000","last_undegraded":"2023-10-14T01:14:24.406282+0000","last_fullsized":"2023-10-14T01:14:24.406282+0000","mapping_epoch":15,"log_start":"87'471433","ondisk_log_start":"87'471433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7216836627,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":480632,"num_read_kb":1891852,"num_write":478901,"num_write_kb":1915598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'13101","reported_seq":17912,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:24.177729+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:14:24.177729+0000","last_peered":"2023-10-14T01:14:24.177729+0000","last_clean":"2023-10-14T01:14:24.177729+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:14:24.177729+0000","last_undegraded":"2023-10-14T01:14:24.177729+0000","last_fullsized":"2023-10-14T01:14:24.177729+0000","mapping_epoch":12,"log_start":"71'3100","ondisk_log_start":"71'3100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:08.956610+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:13:08.956610+0000","last_peered":"2023-10-14T01:13:08.956610+0000","last_clean":"2023-10-14T01:13:08.956610+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:13:08.956610+0000","last_undegraded":"2023-10-14T01:13:08.956610+0000","last_fullsized":"2023-10-14T01:13:08.956610+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28746276883,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1912885,"num_read_kb":7529752,"num_write":1910455,"num_write_kb":7641814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15649865728,"data_stored":18717892646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40085,"ondisk_log_size":40085,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46722,"ondisk_log_size":46722,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9032296,"kb_used_data":7641632,"kb_used_omap":978,"kb_used_meta":1389677,"kb_avail":85339544,"statfs":{"total":96636764160,"available":87387693056,"internally_reserved":0,"allocated":7825031168,"data_stored":9358544280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1423029901},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.38800000000000001},"min":{"1min":0.17000000000000001,"5min":0.16700000000000001,"15min":0.158},"max":{"1min":0.93600000000000005,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.25600000000000001,"5min":0.32800000000000001,"15min":0.35699999999999998},"min":{"1min":0.20899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.33800000000000002,"5min":3.4950000000000001,"15min":5.1109999999999998},"last":0.309}]},{"osd":1,"last update":"Sat Oct 14 01:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.45600000000000002,"15min":0.48399999999999999},"min":{"1min":0.19800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.52100000000000002,"5min":4.1740000000000004,"15min":7.8159999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.501,"15min":0.55100000000000005},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":4.2060000000000004,"15min":16.747},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739210,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5104284,"kb_used_data":3832148,"kb_used_omap":510,"kb_used_meta":1271617,"kb_avail":89267556,"statfs":{"total":96636764160,"available":91409977344,"internally_reserved":0,"allocated":3924119552,"data_stored":4691212567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1302135892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.442,"15min":0.47499999999999998},"min":{"1min":0.18099999999999999,"5min":0.152,"15min":0.152},"max":{"1min":3.7130000000000001,"5min":3.7130000000000001,"15min":5.2370000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.45600000000000002,"15min":0.46899999999999997},"min":{"1min":0.21199999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":8.8089999999999993},"last":0.33000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.20999999999999999,"5min":0.193,"15min":0.193},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.312},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.442,"15min":0.46100000000000002},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.255},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.374}]}]},{"osd":0,"up_from":8,"seq":34359739212,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5025284,"kb_used_data":3812732,"kb_used_omap":502,"kb_used_meta":1212041,"kb_avail":89346556,"statfs":{"total":96636764160,"available":91490873344,"internally_reserved":0,"allocated":3904237568,"data_stored":4669781322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1241130454},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,2,6,11,2,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.436},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":3.8959999999999999,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":5.8600000000000003},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.45200000000000001,"15min":0.434},"min":{"1min":0.191,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":0.77100000000000002,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":6.1520000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.59099999999999997,"15min":0.53400000000000003},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.188},"max":{"1min":3.3330000000000002,"5min":5.843,"15min":5.875},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.41099999999999998,"15min":0.43099999999999999},"min":{"1min":0.27500000000000002,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":3.944,"5min":4.492,"15min":6.5449999999999999},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3902717952,"data_stored":4668903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3922505728,"data_stored":4690333715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7824642048,"data_stored":9358655507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:14:26.839 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:26.839 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:26.840 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:26.840 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:26.841 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:46.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:14:47.356 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:14:47.357 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:14:47.391 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2190,"stamp":"2023-10-14T01:14:46.395726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28778221388,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1942987,"num_read_kb":7579262,"num_write":1918988,"num_write_kb":7674963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87067,"ondisk_log_size":87067,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19403340,"kb_used_data":15347572,"kb_used_omap":1991,"kb_used_meta":4053752,"kb_avail":263712180,"statfs":{"total":289910292480,"available":270041272320,"internally_reserved":0,"allocated":15715913728,"data_stored":18782063609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":4151042231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,7,22,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19525632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4752,"num_read_kb":18968,"num_write":4817,"num_write_kb":19268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002117"},"pg_stats":[{"pgid":"2.7","version":"89'9938","reported_seq":13770,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:41.810308+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:14:41.810308+0000","last_peered":"2023-10-14T01:14:41.810308+0000","last_clean":"2023-10-14T01:14:41.810308+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:14:41.810308+0000","last_undegraded":"2023-10-14T01:14:41.810308+0000","last_fullsized":"2023-10-14T01:14:41.810308+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6693","reported_seq":10201,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:41.809565+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:14:41.809565+0000","last_peered":"2023-10-14T01:14:41.809565+0000","last_clean":"2023-10-14T01:14:41.809565+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:14:41.809565+0000","last_undegraded":"2023-10-14T01:14:41.809565+0000","last_fullsized":"2023-10-14T01:14:41.809565+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6693,"log_dups_size":0,"ondisk_log_size":6693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":173,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:52.124812+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:13:52.124812+0000","last_peered":"2023-10-14T01:13:52.124812+0000","last_clean":"2023-10-14T01:13:52.124812+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:13:52.124812+0000","last_undegraded":"2023-10-14T01:13:52.124812+0000","last_fullsized":"2023-10-14T01:13:52.124812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'13210","reported_seq":18687,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:41.810749+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:14:41.810749+0000","last_peered":"2023-10-14T01:14:41.810749+0000","last_clean":"2023-10-14T01:14:41.810749+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:14:41.810749+0000","last_undegraded":"2023-10-14T01:14:41.810749+0000","last_fullsized":"2023-10-14T01:14:41.810749+0000","mapping_epoch":12,"log_start":"71'3200","ondisk_log_start":"71'3200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'476962","reported_seq":947901,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:43.530641+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:14:43.530641+0000","last_peered":"2023-10-14T01:14:43.530641+0000","last_clean":"2023-10-14T01:14:43.530641+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:14:43.530641+0000","last_undegraded":"2023-10-14T01:14:43.530641+0000","last_fullsized":"2023-10-14T01:14:43.530641+0000","mapping_epoch":15,"log_start":"88'466930","ondisk_log_start":"88'466930","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2047,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 260s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7088635904,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":473133,"num_read_kb":1862468,"num_write":472994,"num_write_kb":1891976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'9826","reported_seq":13545,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:41.809641+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:14:41.809641+0000","last_peered":"2023-10-14T01:14:41.809641+0000","last_clean":"2023-10-14T01:14:41.809641+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:14:41.809641+0000","last_undegraded":"2023-10-14T01:14:41.809641+0000","last_fullsized":"2023-10-14T01:14:41.809641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9826,"log_dups_size":0,"ondisk_log_size":9826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'484501","reported_seq":962540,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:44.516280+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:14:44.516280+0000","last_peered":"2023-10-14T01:14:44.516280+0000","last_clean":"2023-10-14T01:14:44.516280+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:14:44.516280+0000","last_undegraded":"2023-10-14T01:14:44.516280+0000","last_fullsized":"2023-10-14T01:14:44.516280+0000","mapping_epoch":15,"log_start":"88'474427","ondisk_log_start":"88'474427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1747,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 256s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7220293632,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":480293,"num_read_kb":1890320,"num_write":480709,"num_write_kb":1922836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3830,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:41.840015+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:14:41.840015+0000","last_peered":"2023-10-14T01:14:41.840015+0000","last_clean":"2023-10-14T01:14:41.840015+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:14:41.840015+0000","last_undegraded":"2023-10-14T01:14:41.840015+0000","last_fullsized":"2023-10-14T01:14:41.840015+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'487524","reported_seq":969878,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:14:43.531042+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:14:43.531042+0000","last_peered":"2023-10-14T01:14:43.531042+0000","last_clean":"2023-10-14T01:14:43.531042+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:14:43.531042+0000","last_undegraded":"2023-10-14T01:14:43.531042+0000","last_fullsized":"2023-10-14T01:14:43.531042+0000","mapping_epoch":15,"log_start":"88'477427","ondisk_log_start":"88'477427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1807,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 264s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7244181504,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":484613,"num_read_kb":1908208,"num_write":483683,"num_write_kb":1934732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1390,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:36.707496+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:14:36.707496+0000","last_peered":"2023-10-14T01:14:36.707496+0000","last_clean":"2023-10-14T01:14:36.707496+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:14:36.707496+0000","last_undegraded":"2023-10-14T01:14:36.707496+0000","last_fullsized":"2023-10-14T01:14:36.707496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'483364","reported_seq":963541,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:44.545267+0000","last_change":"2023-10-14T01:14:44.446228+0000","last_active":"2023-10-14T01:14:44.545267+0000","last_peered":"2023-10-14T01:14:44.545267+0000","last_clean":"2023-10-14T01:14:44.545267+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:14:44.545267+0000","last_undegraded":"2023-10-14T01:14:44.545267+0000","last_fullsized":"2023-10-14T01:14:44.545267+0000","mapping_epoch":15,"log_start":"88'473333","ondisk_log_start":"88'473333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7224516627,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":482511,"num_read_kb":1899352,"num_write":480792,"num_write_kb":1923162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'13125","reported_seq":17952,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:41.810900+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:14:41.810900+0000","last_peered":"2023-10-14T01:14:41.810900+0000","last_clean":"2023-10-14T01:14:41.810900+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:14:41.810900+0000","last_undegraded":"2023-10-14T01:14:41.810900+0000","last_fullsized":"2023-10-14T01:14:41.810900+0000","mapping_epoch":12,"log_start":"71'3100","ondisk_log_start":"71'3100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:08.956610+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:13:08.956610+0000","last_peered":"2023-10-14T01:13:08.956610+0000","last_clean":"2023-10-14T01:13:08.956610+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:13:08.956610+0000","last_undegraded":"2023-10-14T01:13:08.956610+0000","last_fullsized":"2023-10-14T01:13:08.956610+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28777627667,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1920550,"num_read_kb":7560348,"num_write":1918178,"num_write_kb":7672706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15712911360,"data_stored":18780938278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40234,"ondisk_log_size":40234,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46734,"ondisk_log_size":46734,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739213,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9153368,"kb_used_data":7672528,"kb_used_omap":978,"kb_used_meta":1479853,"kb_avail":85218472,"statfs":{"total":96636764160,"available":87263715328,"internally_reserved":0,"allocated":7856668672,"data_stored":9390181784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1515370125},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.27700000000000002,"5min":0.34000000000000002,"15min":0.374},"min":{"1min":0.18099999999999999,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.48899999999999999,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.248},{"interface":"front","average":{"1min":0.502,"5min":0.34399999999999997,"15min":0.36699999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17199999999999999},"max":{"1min":2.8559999999999999,"5min":2.8559999999999999,"15min":5.1109999999999998},"last":0.35499999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.218,"5min":0.189,"15min":0.189},"max":{"1min":0.84699999999999998,"5min":4.1740000000000004,"15min":4.1840000000000002},"last":0.317},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.499,"15min":0.53600000000000003},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.18099999999999999},"max":{"1min":0.877,"5min":4.2060000000000004,"15min":16.747},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739214,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5173784,"kb_used_data":3847504,"kb_used_omap":510,"kb_used_meta":1325761,"kb_avail":89198056,"statfs":{"total":96636764160,"available":91338809344,"internally_reserved":0,"allocated":3939844096,"data_stored":4706937111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1357579348},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,14],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.442,"15min":0.47499999999999998},"min":{"1min":0.18099999999999999,"5min":0.152,"15min":0.152},"max":{"1min":3.7130000000000001,"5min":3.7130000000000001,"15min":5.2370000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.45600000000000002,"15min":0.46899999999999997},"min":{"1min":0.21199999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":8.8089999999999993},"last":0.317}]},{"osd":2,"last update":"Sat Oct 14 01:13:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.20999999999999999,"5min":0.193,"15min":0.193},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.442,"15min":0.46100000000000002},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.255},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.29999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739216,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5076188,"kb_used_data":3827540,"kb_used_omap":502,"kb_used_meta":1248137,"kb_avail":89295652,"statfs":{"total":96636764160,"available":91438747648,"internally_reserved":0,"allocated":3919400960,"data_stored":4684944714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1278092758},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,8,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47199999999999998,"15min":0.436},"min":{"1min":0.20000000000000001,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":3.8959999999999999,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.45200000000000001,"15min":0.434},"min":{"1min":0.191,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":0.77100000000000002,"5min":4.6420000000000003,"15min":6.5890000000000004},"last":0.216}]},{"osd":2,"last update":"Sat Oct 14 01:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.59099999999999997,"15min":0.53400000000000003},"min":{"1min":0.22900000000000001,"5min":0.20100000000000001,"15min":0.188},"max":{"1min":3.3330000000000002,"5min":5.843,"15min":5.875},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.41099999999999998,"15min":0.43099999999999999},"min":{"1min":0.27500000000000002,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":3.944,"5min":4.492,"15min":6.5449999999999999},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3918553088,"data_stored":4684738560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3938324480,"data_stored":4706152467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7856033792,"data_stored":9390047251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:14:47.393 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:47.394 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:47.394 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:47.394 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:14:47.395 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:14:47.968 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:14:47.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:14:48.490 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:14:48.516 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:14:49.109 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:14:49.140 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:14:49.599 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:14:49.625 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:14:50.414 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:14:50.416+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:14:50.420 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:14:50.420+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:15:09.627 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:10.192 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:15:10.194 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:15:10.234 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2201,"stamp":"2023-10-14T01:15:08.399448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28809183052,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1950407,"num_read_kb":7608850,"num_write":1926601,"num_write_kb":7705415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87269,"ondisk_log_size":87269,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19639156,"kb_used_data":15409052,"kb_used_omap":1991,"kb_used_meta":4228088,"kb_avail":263476364,"statfs":{"total":289910292480,"available":269799796736,"internally_reserved":0,"allocated":15778869248,"data_stored":18845019129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":4329562295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,15,7,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15736832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3766,"num_read_kb":15012,"num_write":3869,"num_write_kb":15476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002112"},"pg_stats":[{"pgid":"2.7","version":"89'9968","reported_seq":13812,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.294421+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:15:02.294421+0000","last_peered":"2023-10-14T01:15:02.294421+0000","last_clean":"2023-10-14T01:15:02.294421+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:15:02.294421+0000","last_undegraded":"2023-10-14T01:15:02.294421+0000","last_fullsized":"2023-10-14T01:15:02.294421+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"89'6713","reported_seq":10229,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.294257+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:15:02.294257+0000","last_peered":"2023-10-14T01:15:02.294257+0000","last_clean":"2023-10-14T01:15:02.294257+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:15:02.294257+0000","last_undegraded":"2023-10-14T01:15:02.294257+0000","last_fullsized":"2023-10-14T01:15:02.294257+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":173,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:52.124812+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:13:52.124812+0000","last_peered":"2023-10-14T01:13:52.124812+0000","last_clean":"2023-10-14T01:13:52.124812+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:13:52.124812+0000","last_undegraded":"2023-10-14T01:13:52.124812+0000","last_fullsized":"2023-10-14T01:13:52.124812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"89'13250","reported_seq":18743,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.294649+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:15:02.294649+0000","last_peered":"2023-10-14T01:15:02.294649+0000","last_clean":"2023-10-14T01:15:02.294649+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:15:02.294649+0000","last_undegraded":"2023-10-14T01:15:02.294649+0000","last_fullsized":"2023-10-14T01:15:02.294649+0000","mapping_epoch":12,"log_start":"71'3200","ondisk_log_start":"71'3200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"89'478986","reported_seq":951756,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:03.753789+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:15:03.753789+0000","last_peered":"2023-10-14T01:15:03.753789+0000","last_clean":"2023-10-14T01:15:03.753789+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:15:03.753789+0000","last_undegraded":"2023-10-14T01:15:03.753789+0000","last_fullsized":"2023-10-14T01:15:03.753789+0000","mapping_epoch":15,"log_start":"88'468930","ondisk_log_start":"88'468930","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2182,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 280s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7096459264,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":474953,"num_read_kb":1869724,"num_write":474923,"num_write_kb":1899692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"89'9880","reported_seq":13611,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.589850+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:15:02.589850+0000","last_peered":"2023-10-14T01:15:02.589850+0000","last_clean":"2023-10-14T01:15:02.589850+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:15:02.589850+0000","last_undegraded":"2023-10-14T01:15:02.589850+0000","last_fullsized":"2023-10-14T01:15:02.589850+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"89'486475","reported_seq":966387,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:04.645333+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:15:04.645333+0000","last_peered":"2023-10-14T01:15:04.645333+0000","last_clean":"2023-10-14T01:15:04.645333+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:15:04.645333+0000","last_undegraded":"2023-10-14T01:15:04.645333+0000","last_fullsized":"2023-10-14T01:15:04.645333+0000","mapping_epoch":15,"log_start":"88'476427","ondisk_log_start":"88'476427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1879,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 276s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7227973632,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":482152,"num_read_kb":1897732,"num_write":482593,"num_write_kb":1930372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3842,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.360381+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:15:02.360381+0000","last_peered":"2023-10-14T01:15:02.360381+0000","last_clean":"2023-10-14T01:15:02.360381+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:15:02.360381+0000","last_undegraded":"2023-10-14T01:15:02.360381+0000","last_fullsized":"2023-10-14T01:15:02.360381+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"89'489577","reported_seq":973811,"reported_epoch":89,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:03.747186+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:15:03.747186+0000","last_peered":"2023-10-14T01:15:03.747186+0000","last_clean":"2023-10-14T01:15:03.747186+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:15:03.747186+0000","last_undegraded":"2023-10-14T01:15:03.747186+0000","last_fullsized":"2023-10-14T01:15:03.747186+0000","mapping_epoch":15,"log_start":"88'479527","ondisk_log_start":"88'479527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1981,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 284s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7252025344,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":486480,"num_read_kb":1915644,"num_write":485616,"num_write_kb":1942464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1394,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:14:57.169854+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:14:57.169854+0000","last_peered":"2023-10-14T01:14:57.169854+0000","last_clean":"2023-10-14T01:14:57.169854+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:14:57.169854+0000","last_undegraded":"2023-10-14T01:14:57.169854+0000","last_fullsized":"2023-10-14T01:14:57.169854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"89'485231","reported_seq":967286,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:04.688348+0000","last_change":"2023-10-14T01:14:59.562435+0000","last_active":"2023-10-14T01:15:04.688348+0000","last_peered":"2023-10-14T01:15:04.688348+0000","last_clean":"2023-10-14T01:15:04.688348+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:15:04.688348+0000","last_undegraded":"2023-10-14T01:15:04.688348+0000","last_fullsized":"2023-10-14T01:15:04.688348+0000","mapping_epoch":15,"log_start":"88'475133","ondisk_log_start":"88'475133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7232131091,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":484385,"num_read_kb":1906836,"num_write":482659,"num_write_kb":1930630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"89'13165","reported_seq":18008,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:02.294846+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:15:02.294846+0000","last_peered":"2023-10-14T01:15:02.294846+0000","last_clean":"2023-10-14T01:15:02.294846+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:15:02.294846+0000","last_undegraded":"2023-10-14T01:15:02.294846+0000","last_fullsized":"2023-10-14T01:15:02.294846+0000","mapping_epoch":12,"log_start":"71'3100","ondisk_log_start":"71'3100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":261,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-14T01:13:08.956610+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:13:08.956610+0000","last_peered":"2023-10-14T01:13:08.956610+0000","last_clean":"2023-10-14T01:13:08.956610+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:13:08.956610+0000","last_undegraded":"2023-10-14T01:13:08.956610+0000","last_fullsized":"2023-10-14T01:13:08.956610+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28808589331,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1927970,"num_read_kb":7589936,"num_write":1925791,"num_write_kb":7703158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15775186944,"data_stored":18843213862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40252,"ondisk_log_size":40252,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":46918,"ondisk_log_size":46918,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739217,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9261724,"kb_used_data":7702996,"kb_used_omap":978,"kb_used_meta":1557741,"kb_avail":85110116,"statfs":{"total":96636764160,"available":87152758784,"internally_reserved":0,"allocated":7887867904,"data_stored":9421381016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1595127437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.27700000000000002,"5min":0.34000000000000002,"15min":0.374},"min":{"1min":0.18099999999999999,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.48899999999999999,"5min":2.5139999999999998,"15min":4.6189999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.34399999999999997,"15min":0.36699999999999999},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.17199999999999999},"max":{"1min":2.8559999999999999,"5min":2.8559999999999999,"15min":5.1109999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:14:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.45500000000000002,"15min":0.45400000000000001},"min":{"1min":0.218,"5min":0.189,"15min":0.189},"max":{"1min":0.84699999999999998,"5min":4.1740000000000004,"15min":4.1840000000000002},"last":0.307},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.499,"15min":0.53600000000000003},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.18099999999999999},"max":{"1min":0.877,"5min":4.2060000000000004,"15min":16.747},"last":0.33500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739218,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5231056,"kb_used_data":3862472,"kb_used_omap":510,"kb_used_meta":1368065,"kb_avail":89140784,"statfs":{"total":96636764160,"available":91280162816,"internally_reserved":0,"allocated":3955171328,"data_stored":4722264343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1400898644},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,8,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41799999999999998,"15min":0.47599999999999998},"min":{"1min":0.17699999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.76000000000000001,"5min":3.7130000000000001,"15min":5.2370000000000001},"last":0.184},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.52200000000000002,"15min":0.49099999999999999},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:15:01 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.23400000000000001,"5min":0.19900000000000001,"15min":0.193},"max":{"1min":0.47799999999999998,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.22},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.436,"15min":0.44900000000000001},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.50600000000000001,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.249}]}]},{"osd":0,"up_from":8,"seq":34359739220,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5146376,"kb_used_data":3843584,"kb_used_omap":502,"kb_used_meta":1302281,"kb_avail":89225464,"statfs":{"total":96636764160,"available":91366875136,"internally_reserved":0,"allocated":3935830016,"data_stored":4701373770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1333536214},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,7,3,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.496,"15min":0.46100000000000002},"min":{"1min":0.191,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":5.8600000000000003,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.224},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.216,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":6.1520000000000001,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.42699999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:14:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.52100000000000002,"15min":0.53300000000000003},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.188},"max":{"1min":0.61499999999999999,"5min":5.843,"15min":5.875},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.38300000000000001,"15min":0.434},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":0.59899999999999998,"5min":3.944,"15min":6.5449999999999999},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3934367744,"data_stored":4700553216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3953684480,"data_stored":4721512467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7887134720,"data_stored":9421148179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:15:10.235 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:10.236 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:10.236 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:10.237 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:10.237 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:16.119 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.119+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:15:16.120 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.119+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:15:16.150 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.151+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:15:16.150 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.151+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:15:16.151 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.151+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:15:16.151 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.151+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:15:16.151 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:15:16.151+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:15:30.239 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:30.803 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:15:30.804 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:15:30.848 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2213,"stamp":"2023-10-14T01:15:30.403121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28847791948,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1959898,"num_read_kb":7646694,"num_write":1936128,"num_write_kb":7743523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87333,"ondisk_log_size":87333,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":19944416,"kb_used_data":15485384,"kb_used_omap":1991,"kb_used_meta":4457016,"kb_avail":263171104,"statfs":{"total":289910292480,"available":269487210496,"internally_reserved":0,"allocated":15857033216,"data_stored":18923170281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":4563984567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,7,12,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23232512,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5747,"num_read_kb":22916,"num_write":5749,"num_write_kb":22996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002028"},"pg_stats":[{"pgid":"2.7","version":"90'10019","reported_seq":13879,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.064599+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:15:29.064599+0000","last_peered":"2023-10-14T01:15:29.064599+0000","last_clean":"2023-10-14T01:15:29.064599+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:15:29.064599+0000","last_undegraded":"2023-10-14T01:15:29.064599+0000","last_fullsized":"2023-10-14T01:15:29.064599+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'6747","reported_seq":10274,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.052715+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:15:29.052715+0000","last_peered":"2023-10-14T01:15:29.052715+0000","last_clean":"2023-10-14T01:15:29.052715+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:15:29.052715+0000","last_undegraded":"2023-10-14T01:15:29.052715+0000","last_fullsized":"2023-10-14T01:15:29.052715+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6747,"log_dups_size":0,"ondisk_log_size":6747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":174,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:59.236697+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:13:59.236697+0000","last_peered":"2023-10-14T01:13:59.236697+0000","last_clean":"2023-10-14T01:13:59.236697+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:13:59.236697+0000","last_undegraded":"2023-10-14T01:13:59.236697+0000","last_fullsized":"2023-10-14T01:13:59.236697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'13318","reported_seq":18832,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.074436+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:15:29.074436+0000","last_peered":"2023-10-14T01:15:29.074436+0000","last_clean":"2023-10-14T01:15:29.074436+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:15:29.074436+0000","last_undegraded":"2023-10-14T01:15:29.074436+0000","last_fullsized":"2023-10-14T01:15:29.074436+0000","mapping_epoch":12,"log_start":"71'3300","ondisk_log_start":"71'3300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'481426","reported_seq":956574,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:28.868931+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:15:28.868931+0000","last_peered":"2023-10-14T01:15:28.868931+0000","last_clean":"2023-10-14T01:15:28.868931+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:15:28.868931+0000","last_undegraded":"2023-10-14T01:15:28.868931+0000","last_fullsized":"2023-10-14T01:15:28.868931+0000","mapping_epoch":15,"log_start":"88'471330","ondisk_log_start":"88'471330","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2377,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 305s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7105810432,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":477311,"num_read_kb":1879112,"num_write":477230,"num_write_kb":1908920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'9946","reported_seq":13693,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.890246+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:15:27.890246+0000","last_peered":"2023-10-14T01:15:27.890246+0000","last_clean":"2023-10-14T01:15:27.890246+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:15:27.890246+0000","last_undegraded":"2023-10-14T01:15:27.890246+0000","last_fullsized":"2023-10-14T01:15:27.890246+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9946,"log_dups_size":0,"ondisk_log_size":9946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'488978","reported_seq":971304,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:29.827432+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:15:29.827432+0000","last_peered":"2023-10-14T01:15:29.827432+0000","last_clean":"2023-10-14T01:15:29.827432+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:15:29.827432+0000","last_undegraded":"2023-10-14T01:15:29.827432+0000","last_fullsized":"2023-10-14T01:15:29.827432+0000","mapping_epoch":15,"log_start":"88'478927","ondisk_log_start":"88'478927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2020,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 301s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7237713920,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":484549,"num_read_kb":1907280,"num_write":484999,"num_write_kb":1939996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3858,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:27.890916+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:15:27.890916+0000","last_peered":"2023-10-14T01:15:27.890916+0000","last_clean":"2023-10-14T01:15:27.890916+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:15:27.890916+0000","last_undegraded":"2023-10-14T01:15:27.890916+0000","last_fullsized":"2023-10-14T01:15:27.890916+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'492144","reported_seq":978751,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:28.836968+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:15:28.836968+0000","last_peered":"2023-10-14T01:15:28.836968+0000","last_clean":"2023-10-14T01:15:28.836968+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:15:28.836968+0000","last_undegraded":"2023-10-14T01:15:28.836968+0000","last_fullsized":"2023-10-14T01:15:28.836968+0000","mapping_epoch":15,"log_start":"88'482127","ondisk_log_start":"88'482127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2155,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 309s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7261954048,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":488841,"num_read_kb":1925076,"num_write":488063,"num_write_kb":1952252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1401,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:22.801247+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:15:22.801247+0000","last_peered":"2023-10-14T01:15:22.801247+0000","last_clean":"2023-10-14T01:15:22.801247+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:15:22.801247+0000","last_undegraded":"2023-10-14T01:15:22.801247+0000","last_fullsized":"2023-10-14T01:15:22.801247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'487598","reported_seq":972042,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.825160+0000","last_change":"2023-10-14T01:15:29.664354+0000","last_active":"2023-10-14T01:15:29.825160+0000","last_peered":"2023-10-14T01:15:29.825160+0000","last_clean":"2023-10-14T01:15:29.825160+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:15:29.825160+0000","last_undegraded":"2023-10-14T01:15:29.825160+0000","last_fullsized":"2023-10-14T01:15:29.825160+0000","mapping_epoch":15,"log_start":"88'477533","ondisk_log_start":"88'477533","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7241719827,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":486760,"num_read_kb":1916312,"num_write":485026,"num_write_kb":1940098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'13233","reported_seq":18097,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:29.081136+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:15:29.081136+0000","last_peered":"2023-10-14T01:15:29.081136+0000","last_clean":"2023-10-14T01:15:29.081136+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:15:29.081136+0000","last_undegraded":"2023-10-14T01:15:29.081136+0000","last_fullsized":"2023-10-14T01:15:29.081136+0000","mapping_epoch":12,"log_start":"71'3200","ondisk_log_start":"71'3200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":263,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:09.762245+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:15:09.762245+0000","last_peered":"2023-10-14T01:15:09.762245+0000","last_clean":"2023-10-14T01:15:09.762245+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:15:09.762245+0000","last_undegraded":"2023-10-14T01:15:09.762245+0000","last_fullsized":"2023-10-14T01:15:09.762245+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28847198227,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1937461,"num_read_kb":7627780,"num_write":1935318,"num_write_kb":7741266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15853248512,"data_stored":18921275430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40229,"ondisk_log_size":40229,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47005,"ondisk_log_size":47005,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739222,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9430232,"kb_used_data":7741136,"kb_used_omap":978,"kb_used_meta":1688109,"kb_avail":84941608,"statfs":{"total":96636764160,"available":86980206592,"internally_reserved":0,"allocated":7926923264,"data_stored":9460432104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1728624269},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.30299999999999999,"15min":0.36899999999999999},"min":{"1min":0.17199999999999999,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.44400000000000001,"5min":0.93600000000000005,"15min":4.6189999999999998},"last":0.247},{"interface":"front","average":{"1min":0.441,"5min":0.35399999999999998,"15min":0.377},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.17199999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.28000000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.67700000000000005,"5min":1.944,"15min":4.1840000000000002},"last":0.307},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.43099999999999999,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":0.71199999999999997,"5min":1.9770000000000001,"15min":16.747},"last":0.33900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739223,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5308460,"kb_used_data":3881572,"kb_used_omap":510,"kb_used_meta":1426369,"kb_avail":89063380,"statfs":{"total":96636764160,"available":91200901120,"internally_reserved":0,"allocated":3974729728,"data_stored":4741818471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1460601940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41799999999999998,"15min":0.47599999999999998},"min":{"1min":0.17699999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.76000000000000001,"5min":3.7130000000000001,"15min":5.2370000000000001},"last":0.191},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.52200000000000002,"15min":0.49099999999999999},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.22700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:15:01 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.23400000000000001,"5min":0.19900000000000001,"15min":0.193},"max":{"1min":0.47799999999999998,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.436,"15min":0.44900000000000001},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.50600000000000001,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.35799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739225,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5205724,"kb_used_data":3862676,"kb_used_omap":502,"kb_used_meta":1342537,"kb_avail":89166116,"statfs":{"total":96636764160,"available":91306102784,"internally_reserved":0,"allocated":3955380224,"data_stored":4720919706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1374758358},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,1,6,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:14:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.496,"15min":0.46100000000000002},"min":{"1min":0.191,"5min":0.17899999999999999,"15min":0.17199999999999999},"max":{"1min":5.8600000000000003,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.216,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":6.1520000000000001,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:14:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.52100000000000002,"15min":0.53300000000000003},"min":{"1min":0.215,"5min":0.20100000000000001,"15min":0.188},"max":{"1min":0.61499999999999999,"5min":5.843,"15min":5.875},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.378,"5min":0.38300000000000001,"15min":0.434},"min":{"1min":0.22700000000000001,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":0.59899999999999998,"5min":3.944,"15min":6.5449999999999999},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3953840128,"data_stored":4720025600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3973218304,"data_stored":4741046291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7926190080,"data_stored":9460203539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:15:30.850 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:30.851 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:30.851 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:30.851 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:30.852 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:15:50.853 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:15:51.445 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:15:51.446 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:15:51.498 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2223,"stamp":"2023-10-14T01:15:50.406663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28878790476,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1967735,"num_read_kb":7677918,"num_write":1943776,"num_write_kb":7774115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87498,"ondisk_log_size":87498,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20203880,"kb_used_data":15546384,"kb_used_omap":1991,"kb_used_meta":4655480,"kb_avail":262911640,"statfs":{"total":289910292480,"available":269221519360,"internally_reserved":0,"allocated":15919497216,"data_stored":18985634281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":4767211703},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,3,6,2,10,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23306240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5852,"num_read_kb":23308,"num_write":5756,"num_write_kb":23024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002162"},"pg_stats":[{"pgid":"2.7","version":"90'10076","reported_seq":13948,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.393700+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:15:48.393700+0000","last_peered":"2023-10-14T01:15:48.393700+0000","last_clean":"2023-10-14T01:15:48.393700+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:15:48.393700+0000","last_undegraded":"2023-10-14T01:15:48.393700+0000","last_fullsized":"2023-10-14T01:15:48.393700+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'6785","reported_seq":10320,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.378791+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:15:48.378791+0000","last_peered":"2023-10-14T01:15:48.378791+0000","last_clean":"2023-10-14T01:15:48.378791+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:15:48.378791+0000","last_undegraded":"2023-10-14T01:15:48.378791+0000","last_fullsized":"2023-10-14T01:15:48.378791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6785,"log_dups_size":0,"ondisk_log_size":6785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":174,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:13:59.236697+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:13:59.236697+0000","last_peered":"2023-10-14T01:13:59.236697+0000","last_clean":"2023-10-14T01:13:59.236697+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:13:59.236697+0000","last_undegraded":"2023-10-14T01:13:59.236697+0000","last_fullsized":"2023-10-14T01:13:59.236697+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'13394","reported_seq":18924,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.388142+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:15:48.388142+0000","last_peered":"2023-10-14T01:15:48.388142+0000","last_clean":"2023-10-14T01:15:48.388142+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:15:48.388142+0000","last_undegraded":"2023-10-14T01:15:48.388142+0000","last_fullsized":"2023-10-14T01:15:48.388142+0000","mapping_epoch":12,"log_start":"71'3300","ondisk_log_start":"71'3300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'483476","reported_seq":960588,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:48.988812+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:15:48.988812+0000","last_peered":"2023-10-14T01:15:48.988812+0000","last_clean":"2023-10-14T01:15:48.988812+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:15:48.988812+0000","last_undegraded":"2023-10-14T01:15:48.988812+0000","last_fullsized":"2023-10-14T01:15:48.988812+0000","mapping_epoch":15,"log_start":"89'473430","ondisk_log_start":"89'473430","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2461,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 325s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7113891840,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":479259,"num_read_kb":1886876,"num_write":479220,"num_write_kb":1916880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'9970","reported_seq":13729,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.373627+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:15:48.373627+0000","last_peered":"2023-10-14T01:15:48.373627+0000","last_clean":"2023-10-14T01:15:48.373627+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:15:48.373627+0000","last_undegraded":"2023-10-14T01:15:48.373627+0000","last_fullsized":"2023-10-14T01:15:48.373627+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":9970,"log_dups_size":0,"ondisk_log_size":9970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'490981","reported_seq":975309,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:49.917023+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:15:49.917023+0000","last_peered":"2023-10-14T01:15:49.917023+0000","last_clean":"2023-10-14T01:15:49.917023+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:15:49.917023+0000","last_undegraded":"2023-10-14T01:15:49.917023+0000","last_fullsized":"2023-10-14T01:15:49.917023+0000","mapping_epoch":15,"log_start":"89'480927","ondisk_log_start":"89'480927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2137,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 322s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7245541376,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":486536,"num_read_kb":1915212,"num_write":486923,"num_write_kb":1947692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3871,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.404644+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:15:48.404644+0000","last_peered":"2023-10-14T01:15:48.404644+0000","last_clean":"2023-10-14T01:15:48.404644+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:15:48.404644+0000","last_undegraded":"2023-10-14T01:15:48.404644+0000","last_fullsized":"2023-10-14T01:15:48.404644+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'494082","reported_seq":982637,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:15:48.991129+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:15:48.991129+0000","last_peered":"2023-10-14T01:15:48.991129+0000","last_clean":"2023-10-14T01:15:48.991129+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:15:48.991129+0000","last_undegraded":"2023-10-14T01:15:48.991129+0000","last_fullsized":"2023-10-14T01:15:48.991129+0000","mapping_epoch":15,"log_start":"89'484027","ondisk_log_start":"89'484027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2308,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 329s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7269371904,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":490769,"num_read_kb":1932752,"num_write":489894,"num_write_kb":1959576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1405,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:43.250261+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:15:43.250261+0000","last_peered":"2023-10-14T01:15:43.250261+0000","last_clean":"2023-10-14T01:15:43.250261+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:15:43.250261+0000","last_undegraded":"2023-10-14T01:15:43.250261+0000","last_fullsized":"2023-10-14T01:15:43.250261+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'489501","reported_seq":975923,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:49.906328+0000","last_change":"2023-10-14T01:15:49.761170+0000","last_active":"2023-10-14T01:15:49.906328+0000","last_peered":"2023-10-14T01:15:49.906328+0000","last_clean":"2023-10-14T01:15:49.906328+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:15:49.906328+0000","last_undegraded":"2023-10-14T01:15:49.906328+0000","last_fullsized":"2023-10-14T01:15:49.906328+0000","mapping_epoch":15,"log_start":"89'479433","ondisk_log_start":"89'479433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7249391635,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":488734,"num_read_kb":1924164,"num_write":486929,"num_write_kb":1947710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'13309","reported_seq":18189,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:48.432502+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:15:48.432502+0000","last_peered":"2023-10-14T01:15:48.432502+0000","last_clean":"2023-10-14T01:15:48.432502+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:15:48.432502+0000","last_undegraded":"2023-10-14T01:15:48.432502+0000","last_fullsized":"2023-10-14T01:15:48.432502+0000","mapping_epoch":12,"log_start":"73'3300","ondisk_log_start":"73'3300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":263,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:09.762245+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:15:09.762245+0000","last_peered":"2023-10-14T01:15:09.762245+0000","last_clean":"2023-10-14T01:15:09.762245+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:15:09.762245+0000","last_undegraded":"2023-10-14T01:15:09.762245+0000","last_fullsized":"2023-10-14T01:15:09.762245+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28878196755,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1945298,"num_read_kb":7659004,"num_write":1942966,"num_write_kb":7771858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15915868160,"data_stored":18983895078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40223,"ondisk_log_size":40223,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47176,"ondisk_log_size":47176,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739226,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9550864,"kb_used_data":7771592,"kb_used_omap":978,"kb_used_meta":1778285,"kb_avail":84820976,"statfs":{"total":96636764160,"available":86856679424,"internally_reserved":0,"allocated":7958110208,"data_stored":9491619048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1820964493},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.30299999999999999,"15min":0.36899999999999999},"min":{"1min":0.17199999999999999,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.44400000000000001,"5min":0.93600000000000005,"15min":4.6189999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.35399999999999998,"15min":0.377},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.17199999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.436}]},{"osd":1,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.67700000000000005,"5min":1.944,"15min":4.1840000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.43099999999999999,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":0.71199999999999997,"5min":1.9770000000000001,"15min":16.747},"last":0.504}]}]},{"osd":1,"up_from":8,"seq":34359739227,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5377900,"kb_used_data":3896868,"kb_used_omap":510,"kb_used_meta":1480513,"kb_avail":88993940,"statfs":{"total":96636764160,"available":91129794560,"internally_reserved":0,"allocated":3990392832,"data_stored":4757481575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1516045396},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,3,1,8,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:01 2023","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.41799999999999998,"15min":0.47599999999999998},"min":{"1min":0.17699999999999999,"5min":0.152,"15min":0.152},"max":{"1min":0.76000000000000001,"5min":3.7130000000000001,"15min":5.2370000000000001},"last":0.188},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.52200000000000002,"15min":0.49099999999999999},"min":{"1min":0.216,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":5.8879999999999999,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.29399999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:15:01 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.38800000000000001,"15min":0.40100000000000002},"min":{"1min":0.23400000000000001,"5min":0.19900000000000001,"15min":0.193},"max":{"1min":0.47799999999999998,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.436,"15min":0.44900000000000001},"min":{"1min":0.29999999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.50600000000000001,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.35699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739229,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5275116,"kb_used_data":3877924,"kb_used_omap":502,"kb_used_meta":1396681,"kb_avail":89096724,"statfs":{"total":96636764160,"available":91235045376,"internally_reserved":0,"allocated":3970994176,"data_stored":4736533658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1430201814},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,2,0,2,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.47599999999999998},"min":{"1min":0.186,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":4.8680000000000003,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.186},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.502,"15min":0.47799999999999998},"min":{"1min":0.193,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":5.9779999999999998,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.219}]},{"osd":2,"last update":"Sat Oct 14 01:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.46200000000000002,"15min":0.52900000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.188},"max":{"1min":1.2470000000000001,"5min":5.843,"15min":5.875},"last":0.25},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38800000000000001,"15min":0.433},"min":{"1min":0.23799999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":0.82199999999999995,"5min":3.944,"15min":6.5449999999999999},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3969507328,"data_stored":4735692800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3988893696,"data_stored":4756721683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7957467136,"data_stored":9491480595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:15:51.500 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:51.500 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:51.500 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:51.501 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:15:51.501 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:11.503 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:12.064 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:16:12.066 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:16:12.096 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2233,"stamp":"2023-10-14T01:16:10.410380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28910006092,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1975247,"num_read_kb":7707846,"num_write":1951474,"num_write_kb":7804907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87500,"ondisk_log_size":87500,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20443272,"kb_used_data":15607728,"kb_used_omap":1991,"kb_used_meta":4833528,"kb_avail":262672248,"statfs":{"total":289910292480,"available":268976381952,"internally_reserved":0,"allocated":15982313472,"data_stored":19048450537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":4949532855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,9,8,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23666688,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5733,"num_read_kb":22832,"num_write":5839,"num_write_kb":23356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002258"},"pg_stats":[{"pgid":"2.7","version":"90'10121","reported_seq":14005,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:10.048763+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:16:10.048763+0000","last_peered":"2023-10-14T01:16:10.048763+0000","last_clean":"2023-10-14T01:16:10.048763+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:16:10.048763+0000","last_undegraded":"2023-10-14T01:16:10.048763+0000","last_fullsized":"2023-10-14T01:16:10.048763+0000","mapping_epoch":12,"log_start":"41'100","ondisk_log_start":"41'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"90'6815","reported_seq":10358,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:10.001120+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:16:10.001120+0000","last_peered":"2023-10-14T01:16:10.001120+0000","last_clean":"2023-10-14T01:16:10.001120+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:16:10.001120+0000","last_undegraded":"2023-10-14T01:16:10.001120+0000","last_fullsized":"2023-10-14T01:16:10.001120+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:12.139350+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:15:12.139350+0000","last_peered":"2023-10-14T01:15:12.139350+0000","last_clean":"2023-10-14T01:15:12.139350+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:15:12.139350+0000","last_undegraded":"2023-10-14T01:15:12.139350+0000","last_fullsized":"2023-10-14T01:15:12.139350+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"90'13454","reported_seq":19001,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:10.053458+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:16:10.053458+0000","last_peered":"2023-10-14T01:16:10.053458+0000","last_clean":"2023-10-14T01:16:10.053458+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:16:10.053458+0000","last_undegraded":"2023-10-14T01:16:10.053458+0000","last_fullsized":"2023-10-14T01:16:10.053458+0000","mapping_epoch":12,"log_start":"73'3400","ondisk_log_start":"73'3400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"90'485480","reported_seq":964440,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:09.192222+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:16:09.192222+0000","last_peered":"2023-10-14T01:16:09.192222+0000","last_clean":"2023-10-14T01:16:09.192222+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:16:09.192222+0000","last_undegraded":"2023-10-14T01:16:09.192222+0000","last_fullsized":"2023-10-14T01:16:09.192222+0000","mapping_epoch":15,"log_start":"89'475430","ondisk_log_start":"89'475430","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2563,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 345s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7121707008,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":481095,"num_read_kb":1894192,"num_write":481150,"num_write_kb":1924600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"90'10009","reported_seq":13781,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:08.849762+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:16:08.849762+0000","last_peered":"2023-10-14T01:16:08.849762+0000","last_clean":"2023-10-14T01:16:08.849762+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:16:08.849762+0000","last_undegraded":"2023-10-14T01:16:08.849762+0000","last_fullsized":"2023-10-14T01:16:08.849762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"90'492973","reported_seq":979202,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:10.073364+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:16:10.073364+0000","last_peered":"2023-10-14T01:16:10.073364+0000","last_clean":"2023-10-14T01:16:10.073364+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:16:10.073364+0000","last_undegraded":"2023-10-14T01:16:10.073364+0000","last_fullsized":"2023-10-14T01:16:10.073364+0000","mapping_epoch":15,"log_start":"89'482927","ondisk_log_start":"89'482927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2269,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 342s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7253237760,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":488430,"num_read_kb":1922760,"num_write":488819,"num_write_kb":1955276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3883,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:08.849762+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:16:08.849762+0000","last_peered":"2023-10-14T01:16:08.849762+0000","last_clean":"2023-10-14T01:16:08.849762+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:16:08.849762+0000","last_undegraded":"2023-10-14T01:16:08.849762+0000","last_fullsized":"2023-10-14T01:16:08.849762+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"90'496059","reported_seq":986546,"reported_epoch":90,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:09.198975+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:16:09.198975+0000","last_peered":"2023-10-14T01:16:09.198975+0000","last_clean":"2023-10-14T01:16:09.198975+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:16:09.198975+0000","last_undegraded":"2023-10-14T01:16:09.198975+0000","last_fullsized":"2023-10-14T01:16:09.198975+0000","mapping_epoch":15,"log_start":"89'486027","ondisk_log_start":"89'486027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2449,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 350s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7276953600,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":492695,"num_read_kb":1940428,"num_write":491771,"num_write_kb":1967084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1409,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:03.729712+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:16:03.729712+0000","last_peered":"2023-10-14T01:16:03.729712+0000","last_clean":"2023-10-14T01:16:03.729712+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:16:03.729712+0000","last_undegraded":"2023-10-14T01:16:03.729712+0000","last_fullsized":"2023-10-14T01:16:03.729712+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"90'491496","reported_seq":979772,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:10.097940+0000","last_change":"2023-10-14T01:16:05.835283+0000","last_active":"2023-10-14T01:16:10.097940+0000","last_peered":"2023-10-14T01:16:10.097940+0000","last_clean":"2023-10-14T01:16:10.097940+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:16:10.097940+0000","last_undegraded":"2023-10-14T01:16:10.097940+0000","last_fullsized":"2023-10-14T01:16:10.097940+0000","mapping_epoch":15,"log_start":"89'481433","ondisk_log_start":"89'481433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7257514003,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":490590,"num_read_kb":1931552,"num_write":488924,"num_write_kb":1955690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"90'13369","reported_seq":18266,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:10.094457+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:16:10.094457+0000","last_peered":"2023-10-14T01:16:10.094457+0000","last_clean":"2023-10-14T01:16:10.094457+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:16:10.094457+0000","last_undegraded":"2023-10-14T01:16:10.094457+0000","last_fullsized":"2023-10-14T01:16:10.094457+0000","mapping_epoch":12,"log_start":"73'3300","ondisk_log_start":"73'3300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":263,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-14T01:15:09.762245+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:15:09.762245+0000","last_peered":"2023-10-14T01:15:09.762245+0000","last_clean":"2023-10-14T01:15:09.762245+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:15:09.762245+0000","last_undegraded":"2023-10-14T01:15:09.762245+0000","last_fullsized":"2023-10-14T01:15:09.762245+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28909412371,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1952810,"num_read_kb":7688932,"num_write":1950664,"num_write_kb":7802650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15978827776,"data_stored":19046854694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40191,"ondisk_log_size":40191,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47210,"ondisk_log_size":47210,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739230,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9669348,"kb_used_data":7802268,"kb_used_omap":978,"kb_used_meta":1866093,"kb_avail":84702492,"statfs":{"total":96636764160,"available":86735351808,"internally_reserved":0,"allocated":7989522432,"data_stored":9523031272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":1910879885},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.30299999999999999,"15min":0.36899999999999999},"min":{"1min":0.17199999999999999,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.44400000000000001,"5min":0.93600000000000005,"15min":4.6189999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.441,"5min":0.35399999999999998,"15min":0.377},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.17199999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.50600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.67700000000000005,"5min":1.944,"15min":4.1840000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.43099999999999999,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":0.71199999999999997,"5min":1.9770000000000001,"15min":16.747},"last":0.60099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739231,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5447528,"kb_used_data":3912352,"kb_used_omap":510,"kb_used_meta":1534657,"kb_avail":88924312,"statfs":{"total":96636764160,"available":91058495488,"internally_reserved":0,"allocated":4006248448,"data_stored":4773337191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1571488852},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42699999999999999,"15min":0.45900000000000002},"min":{"1min":0.16500000000000001,"5min":0.152,"15min":0.152},"max":{"1min":2.492,"5min":3.7130000000000001,"15min":4.117},"last":1.5129999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.20499999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":2.472,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.187}]},{"osd":2,"last update":"Sat Oct 14 01:16:06 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":3.3740000000000001,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46100000000000002,"15min":0.45800000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.4100000000000001,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5326396,"kb_used_data":3893108,"kb_used_omap":502,"kb_used_meta":1432777,"kb_avail":89045444,"statfs":{"total":96636764160,"available":91182534656,"internally_reserved":0,"allocated":3986542592,"data_stored":4752082074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1467164118},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.47599999999999998},"min":{"1min":0.186,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":4.8680000000000003,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.502,"15min":0.47799999999999998},"min":{"1min":0.193,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":5.9779999999999998,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":1.248}]},{"osd":2,"last update":"Sat Oct 14 01:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.46200000000000002,"15min":0.52900000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.188},"max":{"1min":1.2470000000000001,"5min":5.843,"15min":5.875},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38800000000000001,"15min":0.433},"min":{"1min":0.23799999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":0.82199999999999995,"5min":3.944,"15min":6.5449999999999999},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3985096704,"data_stored":4751282176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4004843520,"data_stored":4772671507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7988887552,"data_stored":9522901011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:16:12.098 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:12.098 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:12.099 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:12.099 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:12.099 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:32.101 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:32.575 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:16:32.576 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:16:32.617 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2245,"stamp":"2023-10-14T01:16:32.415823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28941115212,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1982729,"num_read_kb":7737606,"num_write":1959163,"num_write_kb":7835663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87582,"ondisk_log_size":87582,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20705268,"kb_used_data":15668892,"kb_used_omap":1991,"kb_used_meta":5034360,"kb_avail":262410252,"statfs":{"total":289910292480,"available":268708098048,"internally_reserved":0,"allocated":16044945408,"data_stored":19111069528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":5155184823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,2,6,6,8,22,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15446016,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3659,"num_read_kb":14560,"num_write":3815,"num_write_kb":15260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.247381"},"pg_stats":[{"pgid":"2.7","version":"91'10166","reported_seq":14063,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:29.769297+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:16:29.769297+0000","last_peered":"2023-10-14T01:16:29.769297+0000","last_clean":"2023-10-14T01:16:29.769297+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:16:29.769297+0000","last_undegraded":"2023-10-14T01:16:29.769297+0000","last_fullsized":"2023-10-14T01:16:29.769297+0000","mapping_epoch":12,"log_start":"41'100","ondisk_log_start":"41'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'6845","reported_seq":10397,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:29.745865+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:16:29.745865+0000","last_peered":"2023-10-14T01:16:29.745865+0000","last_clean":"2023-10-14T01:16:29.745865+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:16:29.745865+0000","last_undegraded":"2023-10-14T01:16:29.745865+0000","last_fullsized":"2023-10-14T01:16:29.745865+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":176,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:04.995233+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:16:04.995233+0000","last_peered":"2023-10-14T01:16:04.995233+0000","last_clean":"2023-10-14T01:16:04.995233+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:16:04.995233+0000","last_undegraded":"2023-10-14T01:16:04.995233+0000","last_fullsized":"2023-10-14T01:16:04.995233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'13514","reported_seq":19078,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:29.789218+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:16:29.789218+0000","last_peered":"2023-10-14T01:16:29.789218+0000","last_clean":"2023-10-14T01:16:29.789218+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:16:29.789218+0000","last_undegraded":"2023-10-14T01:16:29.789218+0000","last_fullsized":"2023-10-14T01:16:29.789218+0000","mapping_epoch":12,"log_start":"73'3500","ondisk_log_start":"73'3500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'487449","reported_seq":968257,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:29.305193+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:16:29.305193+0000","last_peered":"2023-10-14T01:16:29.305193+0000","last_clean":"2023-10-14T01:16:29.305193+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:16:29.305193+0000","last_undegraded":"2023-10-14T01:16:29.305193+0000","last_fullsized":"2023-10-14T01:16:29.305193+0000","mapping_epoch":15,"log_start":"89'477431","ondisk_log_start":"89'477431","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2719,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 366s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7129260032,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":482934,"num_read_kb":1901492,"num_write":483012,"num_write_kb":1932048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"91'10030","reported_seq":13812,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:29.105271+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:16:29.105271+0000","last_peered":"2023-10-14T01:16:29.105271+0000","last_clean":"2023-10-14T01:16:29.105271+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:16:29.105271+0000","last_undegraded":"2023-10-14T01:16:29.105271+0000","last_fullsized":"2023-10-14T01:16:29.105271+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'494926","reported_seq":983035,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:30.215512+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:16:30.215512+0000","last_peered":"2023-10-14T01:16:30.215512+0000","last_clean":"2023-10-14T01:16:30.215512+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:16:30.215512+0000","last_undegraded":"2023-10-14T01:16:30.215512+0000","last_fullsized":"2023-10-14T01:16:30.215512+0000","mapping_epoch":15,"log_start":"89'484827","ondisk_log_start":"89'484827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2410,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 362s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7260729344,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":490292,"num_read_kb":1930176,"num_write":490675,"num_write_kb":1962700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3893,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:24.332683+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:16:24.332683+0000","last_peered":"2023-10-14T01:16:24.332683+0000","last_clean":"2023-10-14T01:16:24.332683+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:16:24.332683+0000","last_undegraded":"2023-10-14T01:16:24.332683+0000","last_fullsized":"2023-10-14T01:16:24.332683+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'498112","reported_seq":990477,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:29.305690+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:16:29.305690+0000","last_peered":"2023-10-14T01:16:29.305690+0000","last_clean":"2023-10-14T01:16:29.305690+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:16:29.305690+0000","last_undegraded":"2023-10-14T01:16:29.305690+0000","last_fullsized":"2023-10-14T01:16:29.305690+0000","mapping_epoch":15,"log_start":"89'488027","ondisk_log_start":"89'488027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2554,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 370s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7284948992,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":494555,"num_read_kb":1947816,"num_write":493750,"num_write_kb":1975000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1413,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:22.186684+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:16:22.186684+0000","last_peered":"2023-10-14T01:16:22.186684+0000","last_clean":"2023-10-14T01:16:22.186684+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:16:22.186684+0000","last_undegraded":"2023-10-14T01:16:22.186684+0000","last_fullsized":"2023-10-14T01:16:22.186684+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'493488","reported_seq":983696,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:30.214577+0000","last_change":"2023-10-14T01:16:26.842457+0000","last_active":"2023-10-14T01:16:30.214577+0000","last_peered":"2023-10-14T01:16:30.214577+0000","last_clean":"2023-10-14T01:16:30.214577+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:16:30.214577+0000","last_undegraded":"2023-10-14T01:16:30.214577+0000","last_fullsized":"2023-10-14T01:16:30.214577+0000","mapping_epoch":15,"log_start":"89'483433","ondisk_log_start":"89'483433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7265583123,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":492511,"num_read_kb":1939208,"num_write":490916,"num_write_kb":1963658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'13429","reported_seq":18343,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:29.818030+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:16:29.818030+0000","last_peered":"2023-10-14T01:16:29.818030+0000","last_clean":"2023-10-14T01:16:29.818030+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:16:29.818030+0000","last_undegraded":"2023-10-14T01:16:29.818030+0000","last_fullsized":"2023-10-14T01:16:29.818030+0000","mapping_epoch":12,"log_start":"73'3400","ondisk_log_start":"73'3400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":264,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:12.161645+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:15:12.161645+0000","last_peered":"2023-10-14T01:15:12.161645+0000","last_clean":"2023-10-14T01:15:12.161645+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:15:12.161645+0000","last_undegraded":"2023-10-14T01:15:12.161645+0000","last_fullsized":"2023-10-14T01:15:12.161645+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28940521491,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1960292,"num_read_kb":7718692,"num_write":1958353,"num_write_kb":7833406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16041689088,"data_stored":19109716006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40257,"ondisk_log_size":40257,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47226,"ondisk_log_size":47226,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739234,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9804024,"kb_used_data":7832880,"kb_used_omap":978,"kb_used_meta":1970157,"kb_avail":84567816,"statfs":{"total":96636764160,"available":86597443584,"internally_reserved":0,"allocated":8020869120,"data_stored":9554373645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":2017441421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.30299999999999999,"15min":0.36899999999999999},"min":{"1min":0.17199999999999999,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.44400000000000001,"5min":0.93600000000000005,"15min":4.6189999999999998},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.35399999999999998,"15min":0.377},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.17199999999999999},"max":{"1min":3.7519999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38700000000000001,"15min":0.45300000000000001},"min":{"1min":0.20399999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.67700000000000005,"5min":1.944,"15min":4.1840000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.43099999999999999,"15min":0.53800000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":0.71199999999999997,"5min":1.9770000000000001,"15min":16.747},"last":0.39000000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739235,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5498916,"kb_used_data":3927644,"kb_used_omap":510,"kb_used_meta":1570753,"kb_avail":88872924,"statfs":{"total":96636764160,"available":91005874176,"internally_reserved":0,"allocated":4021907456,"data_stored":4788991884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1608451156},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,4,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42699999999999999,"15min":0.45900000000000002},"min":{"1min":0.16500000000000001,"5min":0.152,"15min":0.152},"max":{"1min":2.492,"5min":3.7130000000000001,"15min":4.117},"last":0.25},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.20499999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":2.472,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:16:06 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":3.3740000000000001,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46100000000000002,"15min":0.45800000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.4100000000000001,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.33900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5402328,"kb_used_data":3908368,"kb_used_omap":502,"kb_used_meta":1493449,"kb_avail":88969512,"statfs":{"total":96636764160,"available":91104780288,"internally_reserved":0,"allocated":4002168832,"data_stored":4767703999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1529292246},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,1,1,2,2,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.47599999999999998},"min":{"1min":0.186,"5min":0.185,"15min":0.17199999999999999},"max":{"1min":4.8680000000000003,"5min":6.1130000000000004,"15min":8.3239999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.502,"15min":0.47799999999999998},"min":{"1min":0.193,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":5.9779999999999998,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:15:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.46200000000000002,"15min":0.52900000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.188},"max":{"1min":1.2470000000000001,"5min":5.843,"15min":5.875},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38800000000000001,"15min":0.433},"min":{"1min":0.23799999999999999,"5min":0.21299999999999999,"15min":0.20999999999999999},"max":{"1min":0.82199999999999995,"5min":3.944,"15min":6.5449999999999999},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4000817152,"data_stored":4767002624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4020592640,"data_stored":4788420627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8020279296,"data_stored":9554292755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:16:32.619 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:32.619 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:32.620 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:32.620 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:32.621 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:16:52.622 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:16:53.272 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:16:53.273 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:16:53.310 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2255,"stamp":"2023-10-14T01:16:52.422840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28972142412,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1990044,"num_read_kb":7766750,"num_write":1966824,"num_write_kb":7866307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87701,"ondisk_log_size":87701,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20965780,"kb_used_data":15729980,"kb_used_omap":1991,"kb_used_meta":5233784,"kb_avail":262149740,"statfs":{"total":289910292480,"available":268441333760,"internally_reserved":0,"allocated":16107499520,"data_stored":19173623640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":5359394999},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,10,5,9,14,19,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15478784,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3622,"num_read_kb":14436,"num_write":3820,"num_write_kb":15280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005711"},"pg_stats":[{"pgid":"2.7","version":"91'10202","reported_seq":14112,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:49.814229+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:16:49.814229+0000","last_peered":"2023-10-14T01:16:49.814229+0000","last_clean":"2023-10-14T01:16:49.814229+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:16:49.814229+0000","last_undegraded":"2023-10-14T01:16:49.814229+0000","last_fullsized":"2023-10-14T01:16:49.814229+0000","mapping_epoch":12,"log_start":"43'200","ondisk_log_start":"43'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'6869","reported_seq":10430,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:49.813604+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:16:49.813604+0000","last_peered":"2023-10-14T01:16:49.813604+0000","last_clean":"2023-10-14T01:16:49.813604+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:16:49.813604+0000","last_undegraded":"2023-10-14T01:16:49.813604+0000","last_fullsized":"2023-10-14T01:16:49.813604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6869,"log_dups_size":0,"ondisk_log_size":6869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":176,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:04.995233+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:16:04.995233+0000","last_peered":"2023-10-14T01:16:04.995233+0000","last_clean":"2023-10-14T01:16:04.995233+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:16:04.995233+0000","last_undegraded":"2023-10-14T01:16:04.995233+0000","last_fullsized":"2023-10-14T01:16:04.995233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'13562","reported_seq":19142,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:49.814597+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:16:49.814597+0000","last_peered":"2023-10-14T01:16:49.814597+0000","last_clean":"2023-10-14T01:16:49.814597+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:16:49.814597+0000","last_undegraded":"2023-10-14T01:16:49.814597+0000","last_fullsized":"2023-10-14T01:16:49.814597+0000","mapping_epoch":12,"log_start":"73'3500","ondisk_log_start":"73'3500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'489500","reported_seq":972150,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:49.417126+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:16:49.417126+0000","last_peered":"2023-10-14T01:16:49.417126+0000","last_clean":"2023-10-14T01:16:49.417126+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:16:49.417126+0000","last_undegraded":"2023-10-14T01:16:49.417126+0000","last_fullsized":"2023-10-14T01:16:49.417126+0000","mapping_epoch":15,"log_start":"89'479431","ondisk_log_start":"89'479431","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2842,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 386s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7137222656,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":484760,"num_read_kb":1908776,"num_write":484978,"num_write_kb":1939912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"91'10063","reported_seq":13857,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:47.521604+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:16:47.521604+0000","last_peered":"2023-10-14T01:16:47.521604+0000","last_clean":"2023-10-14T01:16:47.521604+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:16:47.521604+0000","last_undegraded":"2023-10-14T01:16:47.521604+0000","last_fullsized":"2023-10-14T01:16:47.521604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'496963","reported_seq":986902,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:50.279021+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:16:50.279021+0000","last_peered":"2023-10-14T01:16:50.279021+0000","last_clean":"2023-10-14T01:16:50.279021+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:16:50.279021+0000","last_undegraded":"2023-10-14T01:16:50.279021+0000","last_fullsized":"2023-10-14T01:16:50.279021+0000","mapping_epoch":15,"log_start":"89'486927","ondisk_log_start":"89'486927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2524,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 382s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7268663296,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":492110,"num_read_kb":1937428,"num_write":492631,"num_write_kb":1970524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3905,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:44.692604+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:16:44.692604+0000","last_peered":"2023-10-14T01:16:44.692604+0000","last_clean":"2023-10-14T01:16:44.692604+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:16:44.692604+0000","last_undegraded":"2023-10-14T01:16:44.692604+0000","last_fullsized":"2023-10-14T01:16:44.692604+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'500117","reported_seq":994286,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:16:49.415366+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:16:49.415366+0000","last_peered":"2023-10-14T01:16:49.415366+0000","last_clean":"2023-10-14T01:16:49.415366+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:16:49.415366+0000","last_undegraded":"2023-10-14T01:16:49.415366+0000","last_fullsized":"2023-10-14T01:16:49.415366+0000","mapping_epoch":15,"log_start":"89'490027","ondisk_log_start":"89'490027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2704,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 390s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7292678144,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":496345,"num_read_kb":1954956,"num_write":495652,"num_write_kb":1982608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1417,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:39.570251+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:16:39.570251+0000","last_peered":"2023-10-14T01:16:39.570251+0000","last_clean":"2023-10-14T01:16:39.570251+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:16:39.570251+0000","last_undegraded":"2023-10-14T01:16:39.570251+0000","last_fullsized":"2023-10-14T01:16:39.570251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'495325","reported_seq":987424,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:50.281727+0000","last_change":"2023-10-14T01:16:46.804759+0000","last_active":"2023-10-14T01:16:50.281727+0000","last_peered":"2023-10-14T01:16:50.281727+0000","last_clean":"2023-10-14T01:16:50.281727+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:16:50.281727+0000","last_undegraded":"2023-10-14T01:16:50.281727+0000","last_fullsized":"2023-10-14T01:16:50.281727+0000","mapping_epoch":15,"log_start":"89'485233","ondisk_log_start":"89'485233","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7272984595,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":494392,"num_read_kb":1946676,"num_write":492753,"num_write_kb":1971006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'13477","reported_seq":18407,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:49.814876+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:16:49.814876+0000","last_peered":"2023-10-14T01:16:49.814876+0000","last_clean":"2023-10-14T01:16:49.814876+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:16:49.814876+0000","last_undegraded":"2023-10-14T01:16:49.814876+0000","last_fullsized":"2023-10-14T01:16:49.814876+0000","mapping_epoch":12,"log_start":"73'3400","ondisk_log_start":"73'3400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":264,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:15:12.161645+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:15:12.161645+0000","last_peered":"2023-10-14T01:15:12.161645+0000","last_clean":"2023-10-14T01:15:12.161645+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:15:12.161645+0000","last_undegraded":"2023-10-14T01:15:12.161645+0000","last_fullsized":"2023-10-14T01:15:12.161645+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":28971548691,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1967607,"num_read_kb":7747836,"num_write":1966014,"num_write_kb":7864050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16104337408,"data_stored":19172364326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40287,"ondisk_log_size":40287,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47315,"ondisk_log_size":47315,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739238,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9937256,"kb_used_data":7863456,"kb_used_omap":978,"kb_used_meta":2072813,"kb_avail":84434584,"statfs":{"total":96636764160,"available":86461014016,"internally_reserved":0,"allocated":8052178944,"data_stored":9585683469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":2122561165},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.313,"15min":0.34000000000000002},"min":{"1min":0.20200000000000001,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.60999999999999999,"5min":0.93600000000000005,"15min":2.5139999999999998},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.312,"5min":0.35399999999999998,"15min":0.36499999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":0.50600000000000001,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40300000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":1.899,"5min":1.944,"15min":4.1840000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.44700000000000001,"15min":0.55000000000000004},"min":{"1min":0.28599999999999998,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":1.9350000000000001,"5min":1.9770000000000001,"15min":16.747},"last":0.37}]}]},{"osd":1,"up_from":8,"seq":34359739239,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5568204,"kb_used_data":3942788,"kb_used_omap":510,"kb_used_meta":1624897,"kb_avail":88803636,"statfs":{"total":96636764160,"available":90934923264,"internally_reserved":0,"allocated":4037414912,"data_stored":4804499340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1663894612},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:06 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42699999999999999,"15min":0.45900000000000002},"min":{"1min":0.16500000000000001,"5min":0.152,"15min":0.152},"max":{"1min":2.492,"5min":3.7130000000000001,"15min":4.117},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.20499999999999999,"5min":0.17799999999999999,"15min":0.16200000000000001},"max":{"1min":2.472,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":3.2440000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:16:06 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.218,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":3.3740000000000001,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.46100000000000002,"15min":0.45800000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.4100000000000001,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.52400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739241,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5460320,"kb_used_data":3923736,"kb_used_omap":502,"kb_used_meta":1536073,"kb_avail":88911520,"statfs":{"total":96636764160,"available":91045396480,"internally_reserved":0,"allocated":4017905664,"data_stored":4783440831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1572939222},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,0,3,9,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.193,"5min":0.186,"15min":0.17199999999999999},"max":{"1min":0.93400000000000005,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.623},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.56000000000000005,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":3.9870000000000001,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.218}]},{"osd":2,"last update":"Sat Oct 14 01:16:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":0.71799999999999997,"15min":0.62},"min":{"1min":0.21099999999999999,"5min":0.20999999999999999,"15min":0.188},"max":{"1min":16.763999999999999,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.505},{"interface":"front","average":{"1min":1.3260000000000001,"5min":0.58599999999999997,"15min":0.5},"min":{"1min":0.25,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":16.748000000000001,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.66200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4016660480,"data_stored":4782845952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4036132864,"data_stored":4803960851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8051544064,"data_stored":9585557523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:16:53.312 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:53.313 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:53.313 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:53.314 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:16:53.314 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:16:53.784 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:16:53.816 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:16:54.383 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:16:54.418 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:16:54.976 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:16:55.010 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:16:55.496 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:16:55.529 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:15.531 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:15.995 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:17:15.996 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:17:16.028 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2266,"stamp":"2023-10-14T01:17:14.426430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29003050828,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":1997412,"num_read_kb":7796142,"num_write":1974466,"num_write_kb":7896875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87631,"ondisk_log_size":87631,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21230052,"kb_used_data":15798476,"kb_used_omap":1991,"kb_used_meta":5429560,"kb_avail":261885468,"statfs":{"total":289910292480,"available":268170719232,"internally_reserved":0,"allocated":16177639424,"data_stored":19243763544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2039625,"internal_metadata":5559869623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,10,10,8,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15597568,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3730,"num_read_kb":14888,"num_write":3855,"num_write_kb":15420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002017"},"pg_stats":[{"pgid":"2.7","version":"91'10226","reported_seq":14148,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.356062+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:17:10.356062+0000","last_peered":"2023-10-14T01:17:10.356062+0000","last_clean":"2023-10-14T01:17:10.356062+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:17:10.356062+0000","last_undegraded":"2023-10-14T01:17:10.356062+0000","last_fullsized":"2023-10-14T01:17:10.356062+0000","mapping_epoch":12,"log_start":"43'200","ondisk_log_start":"43'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'6885","reported_seq":10454,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.345360+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:17:10.345360+0000","last_peered":"2023-10-14T01:17:10.345360+0000","last_clean":"2023-10-14T01:17:10.345360+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:17:10.345360+0000","last_undegraded":"2023-10-14T01:17:10.345360+0000","last_fullsized":"2023-10-14T01:17:10.345360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":176,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-14T01:16:04.995233+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:16:04.995233+0000","last_peered":"2023-10-14T01:16:04.995233+0000","last_clean":"2023-10-14T01:16:04.995233+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:16:04.995233+0000","last_undegraded":"2023-10-14T01:16:04.995233+0000","last_fullsized":"2023-10-14T01:16:04.995233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"91'13594","reported_seq":19190,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.367400+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:17:10.367400+0000","last_peered":"2023-10-14T01:17:10.367400+0000","last_clean":"2023-10-14T01:17:10.367400+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:17:10.367400+0000","last_undegraded":"2023-10-14T01:17:10.367400+0000","last_fullsized":"2023-10-14T01:17:10.367400+0000","mapping_epoch":12,"log_start":"73'3500","ondisk_log_start":"73'3500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"91'491492","reported_seq":975938,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:09.576084+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:17:09.576084+0000","last_peered":"2023-10-14T01:17:09.576084+0000","last_clean":"2023-10-14T01:17:09.576084+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:17:09.576084+0000","last_undegraded":"2023-10-14T01:17:09.576084+0000","last_fullsized":"2023-10-14T01:17:09.576084+0000","mapping_epoch":15,"log_start":"90'481431","ondisk_log_start":"90'481431","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2920,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 406s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7145070592,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":486547,"num_read_kb":1915908,"num_write":486916,"num_write_kb":1947664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"91'10102","reported_seq":13908,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:09.367868+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:17:09.367868+0000","last_peered":"2023-10-14T01:17:09.367868+0000","last_clean":"2023-10-14T01:17:09.367868+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:17:09.367868+0000","last_undegraded":"2023-10-14T01:17:09.367868+0000","last_fullsized":"2023-10-14T01:17:09.367868+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"91'498899","reported_seq":990733,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:10.330055+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:17:10.330055+0000","last_peered":"2023-10-14T01:17:10.330055+0000","last_clean":"2023-10-14T01:17:10.330055+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:17:10.330055+0000","last_undegraded":"2023-10-14T01:17:10.330055+0000","last_fullsized":"2023-10-14T01:17:10.330055+0000","mapping_epoch":15,"log_start":"90'488827","ondisk_log_start":"90'488827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2668,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 402s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7276077056,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":493990,"num_read_kb":1944932,"num_write":494464,"num_write_kb":1977856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3920,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.291861+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:17:10.291861+0000","last_peered":"2023-10-14T01:17:10.291861+0000","last_clean":"2023-10-14T01:17:10.291861+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:17:10.291861+0000","last_undegraded":"2023-10-14T01:17:10.291861+0000","last_fullsized":"2023-10-14T01:17:10.291861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"91'502173","reported_seq":998182,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:09.588904+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:17:09.588904+0000","last_peered":"2023-10-14T01:17:09.588904+0000","last_clean":"2023-10-14T01:17:09.588904+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:17:09.588904+0000","last_undegraded":"2023-10-14T01:17:09.588904+0000","last_fullsized":"2023-10-14T01:17:09.588904+0000","mapping_epoch":15,"log_start":"90'492127","ondisk_log_start":"90'492127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2827,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 410s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7300620288,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":498175,"num_read_kb":1962252,"num_write":497620,"num_write_kb":1990480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1423,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.290003+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:17:10.290003+0000","last_peered":"2023-10-14T01:17:10.290003+0000","last_clean":"2023-10-14T01:17:10.290003+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:17:10.290003+0000","last_undegraded":"2023-10-14T01:17:10.290003+0000","last_fullsized":"2023-10-14T01:17:10.290003+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"91'497228","reported_seq":991208,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.368372+0000","last_change":"2023-10-14T01:17:06.862372+0000","last_active":"2023-10-14T01:17:10.368372+0000","last_peered":"2023-10-14T01:17:10.368372+0000","last_clean":"2023-10-14T01:17:10.368372+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:17:10.368372+0000","last_undegraded":"2023-10-14T01:17:10.368372+0000","last_fullsized":"2023-10-14T01:17:10.368372+0000","mapping_epoch":15,"log_start":"90'487133","ondisk_log_start":"90'487133","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7280689171,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":496263,"num_read_kb":1954136,"num_write":494656,"num_write_kb":1978618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'13509","reported_seq":18453,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.375839+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:17:10.375839+0000","last_peered":"2023-10-14T01:17:10.375839+0000","last_clean":"2023-10-14T01:17:10.375839+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:17:10.375839+0000","last_undegraded":"2023-10-14T01:17:10.375839+0000","last_fullsized":"2023-10-14T01:17:10.375839+0000","mapping_epoch":12,"log_start":"75'3500","ondisk_log_start":"75'3500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":265,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:22.175731+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:16:22.175731+0000","last_peered":"2023-10-14T01:16:22.175731+0000","last_clean":"2023-10-14T01:16:22.175731+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:16:22.175731+0000","last_undegraded":"2023-10-14T01:16:22.175731+0000","last_fullsized":"2023-10-14T01:16:22.175731+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29002457107,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1974975,"num_read_kb":7777228,"num_write":1973656,"num_write_kb":7894618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16174579712,"data_stored":19242606630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40274,"ondisk_log_size":40274,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47258,"ondisk_log_size":47258,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739243,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10080812,"kb_used_data":7901476,"kb_used_omap":978,"kb_used_meta":2178349,"kb_avail":84291028,"statfs":{"total":96636764160,"available":86314012672,"internally_reserved":0,"allocated":8091111424,"data_stored":9624615949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1001843,"internal_metadata":2230630029},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.313,"15min":0.34000000000000002},"min":{"1min":0.20200000000000001,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":0.60999999999999999,"5min":0.93600000000000005,"15min":2.5139999999999998},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.312,"5min":0.35399999999999998,"15min":0.36499999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":0.50600000000000001,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.22900000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40300000000000002,"15min":0.45600000000000002},"min":{"1min":0.252,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":1.899,"5min":1.944,"15min":4.1840000000000002},"last":3.1800000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.44700000000000001,"15min":0.55000000000000004},"min":{"1min":0.28599999999999998,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":1.9350000000000001,"5min":1.9770000000000001,"15min":16.747},"last":3.2120000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739243,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5619304,"kb_used_data":3957792,"kb_used_omap":510,"kb_used_meta":1660993,"kb_avail":88752536,"statfs":{"total":96636764160,"available":90882596864,"internally_reserved":0,"allocated":4052779008,"data_stored":4819863436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1700856916},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.222,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":7.2809999999999997,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51900000000000002,"15min":0.48999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":3.2440000000000002,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.25800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.40000000000000002,"15min":0.39500000000000002},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.193},"max":{"1min":0.54800000000000004,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.442},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.57499999999999996,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359739245,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5529936,"kb_used_data":3939208,"kb_used_omap":502,"kb_used_meta":1590217,"kb_avail":88841904,"statfs":{"total":96636764160,"available":90974109696,"internally_reserved":0,"allocated":4033748992,"data_stored":4799284159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1628382678},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,7,5,21],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.193,"5min":0.186,"15min":0.17199999999999999},"max":{"1min":0.93400000000000005,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.56000000000000005,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":3.9870000000000001,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.374}]},{"osd":2,"last update":"Sat Oct 14 01:16:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":0.71799999999999997,"15min":0.62},"min":{"1min":0.21099999999999999,"5min":0.20999999999999999,"15min":0.188},"max":{"1min":16.763999999999999,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.308},{"interface":"front","average":{"1min":1.3260000000000001,"5min":0.58599999999999997,"15min":0.5},"min":{"1min":0.25,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":16.748000000000001,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.22600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4032667648,"data_stored":4798853120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4051435520,"data_stored":4819263507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8090476544,"data_stored":9624490003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:17:16.029 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:16.030 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:16.030 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:16.031 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:16.031 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:26.301 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:17:26.296+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:17:36.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:36.569 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:17:36.569 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:17:36.596 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2278,"stamp":"2023-10-14T01:17:36.430089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29041196876,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2006667,"num_read_kb":7833066,"num_write":1983911,"num_write_kb":7934655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87765,"ondisk_log_size":87765,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21126820,"kb_used_data":15866112,"kb_used_omap":2282,"kb_used_meta":5258389,"kb_avail":261988700,"statfs":{"total":289910292480,"available":268276428800,"internally_reserved":0,"allocated":16246898688,"data_stored":19313009734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":5384591220},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,5,11,9,18,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15556608,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3833,"num_read_kb":15300,"num_write":3852,"num_write_kb":15408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001620"},"pg_stats":[{"pgid":"2.7","version":"92'10292","reported_seq":14227,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:34.608482+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:17:34.608482+0000","last_peered":"2023-10-14T01:17:34.608482+0000","last_clean":"2023-10-14T01:17:34.608482+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:17:34.608482+0000","last_undegraded":"2023-10-14T01:17:34.608482+0000","last_fullsized":"2023-10-14T01:17:34.608482+0000","mapping_epoch":12,"log_start":"43'200","ondisk_log_start":"43'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6929","reported_seq":10507,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:34.600789+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:17:34.600789+0000","last_peered":"2023-10-14T01:17:34.600789+0000","last_clean":"2023-10-14T01:17:34.600789+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:17:34.600789+0000","last_undegraded":"2023-10-14T01:17:34.600789+0000","last_fullsized":"2023-10-14T01:17:34.600789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":177,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:22.174754+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:16:22.174754+0000","last_peered":"2023-10-14T01:16:22.174754+0000","last_clean":"2023-10-14T01:16:22.174754+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:16:22.174754+0000","last_undegraded":"2023-10-14T01:16:22.174754+0000","last_fullsized":"2023-10-14T01:16:22.174754+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'13682","reported_seq":19295,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:34.614993+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:17:34.614993+0000","last_peered":"2023-10-14T01:17:34.614993+0000","last_clean":"2023-10-14T01:17:34.614993+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:17:34.614993+0000","last_undegraded":"2023-10-14T01:17:34.614993+0000","last_fullsized":"2023-10-14T01:17:34.614993+0000","mapping_epoch":12,"log_start":"75'3600","ondisk_log_start":"75'3600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'493953","reported_seq":980722,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:34.690467+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:17:34.690467+0000","last_peered":"2023-10-14T01:17:34.690467+0000","last_clean":"2023-10-14T01:17:34.690467+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:17:34.690467+0000","last_undegraded":"2023-10-14T01:17:34.690467+0000","last_fullsized":"2023-10-14T01:17:34.690467+0000","mapping_epoch":15,"log_start":"90'483932","ondisk_log_start":"90'483932","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3106,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 431s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7154401280,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":488851,"num_read_kb":1925096,"num_write":489246,"num_write_kb":1956984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'10148","reported_seq":13970,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:33.773034+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:17:33.773034+0000","last_peered":"2023-10-14T01:17:33.773034+0000","last_clean":"2023-10-14T01:17:33.773034+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:17:33.773034+0000","last_undegraded":"2023-10-14T01:17:33.773034+0000","last_fullsized":"2023-10-14T01:17:33.773034+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'501339","reported_seq":995566,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:35.509847+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:17:35.509847+0000","last_peered":"2023-10-14T01:17:35.509847+0000","last_clean":"2023-10-14T01:17:35.509847+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:17:35.509847+0000","last_undegraded":"2023-10-14T01:17:35.509847+0000","last_fullsized":"2023-10-14T01:17:35.509847+0000","mapping_epoch":15,"log_start":"90'491327","ondisk_log_start":"90'491327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2830,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 427s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7285497856,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":496366,"num_read_kb":1954408,"num_write":496792,"num_write_kb":1987168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3933,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:30.820828+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:17:30.820828+0000","last_peered":"2023-10-14T01:17:30.820828+0000","last_clean":"2023-10-14T01:17:30.820828+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:17:30.820828+0000","last_undegraded":"2023-10-14T01:17:30.820828+0000","last_fullsized":"2023-10-14T01:17:30.820828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'504711","reported_seq":1003093,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:34.677239+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:17:34.677239+0000","last_peered":"2023-10-14T01:17:34.677239+0000","last_clean":"2023-10-14T01:17:34.677239+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:17:34.677239+0000","last_undegraded":"2023-10-14T01:17:34.677239+0000","last_fullsized":"2023-10-14T01:17:34.677239+0000","mapping_epoch":15,"log_start":"90'494627","ondisk_log_start":"90'494627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2989,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 435s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7310467072,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":500531,"num_read_kb":1971656,"num_write":500043,"num_write_kb":2000172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1428,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:30.774100+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:17:30.774100+0000","last_peered":"2023-10-14T01:17:30.774100+0000","last_clean":"2023-10-14T01:17:30.774100+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:17:30.774100+0000","last_undegraded":"2023-10-14T01:17:30.774100+0000","last_fullsized":"2023-10-14T01:17:30.774100+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'499592","reported_seq":995799,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:35.519654+0000","last_change":"2023-10-14T01:17:31.935789+0000","last_active":"2023-10-14T01:17:35.519654+0000","last_peered":"2023-10-14T01:17:35.519654+0000","last_clean":"2023-10-14T01:17:35.519654+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:17:35.519654+0000","last_undegraded":"2023-10-14T01:17:35.519654+0000","last_fullsized":"2023-10-14T01:17:35.519654+0000","mapping_epoch":15,"log_start":"90'489533","ondisk_log_start":"90'489533","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7290236947,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":498482,"num_read_kb":1962992,"num_write":497020,"num_write_kb":1988074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'13597","reported_seq":18560,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:34.639905+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:17:34.639905+0000","last_peered":"2023-10-14T01:17:34.639905+0000","last_clean":"2023-10-14T01:17:34.639905+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:17:34.639905+0000","last_undegraded":"2023-10-14T01:17:34.639905+0000","last_fullsized":"2023-10-14T01:17:34.639905+0000","mapping_epoch":12,"log_start":"75'3500","ondisk_log_start":"75'3500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.378816+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:17:10.378816+0000","last_peered":"2023-10-14T01:17:10.378816+0000","last_clean":"2023-10-14T01:17:10.378816+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:17:10.378816+0000","last_undegraded":"2023-10-14T01:17:10.378816+0000","last_fullsized":"2023-10-14T01:17:10.378816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29040603155,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1984230,"num_read_kb":7814152,"num_write":1983101,"num_write_kb":7932398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16244346880,"data_stored":19312373798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40176,"ondisk_log_size":40176,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47490,"ondisk_log_size":47490,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739247,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9853312,"kb_used_data":7931764,"kb_used_omap":1268,"kb_used_meta":1920267,"kb_avail":84518528,"statfs":{"total":96636764160,"available":86546972672,"internally_reserved":0,"allocated":8122126336,"data_stored":9655626503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":1966353738},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.314,"15min":0.33300000000000002},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":1.155,"5min":1.155,"15min":2.5139999999999998},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":0.36399999999999999,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.221}]},{"osd":1,"last update":"Sat Oct 14 01:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.44600000000000001,"15min":0.46100000000000002},"min":{"1min":0.24099999999999999,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":3.883,"5min":3.883,"15min":4.1840000000000002},"last":3.883},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.49099999999999999,"15min":0.57699999999999996},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":3.915,"5min":3.915,"15min":16.747},"last":3.915}]}]},{"osd":1,"up_from":8,"seq":34359739248,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5684452,"kb_used_data":3976280,"kb_used_omap":510,"kb_used_meta":1707649,"kb_avail":88687388,"statfs":{"total":96636764160,"available":90815885312,"internally_reserved":0,"allocated":4071710720,"data_stored":4838790790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1748632660},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.222,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":7.2809999999999997,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51900000000000002,"15min":0.48999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":3.2440000000000002,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.40000000000000002,"15min":0.39500000000000002},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.193},"max":{"1min":0.54800000000000004,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.442},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.57499999999999996,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739250,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5589056,"kb_used_data":3958068,"kb_used_omap":502,"kb_used_meta":1630473,"kb_avail":88782784,"statfs":{"total":96636764160,"available":90913570816,"internally_reserved":0,"allocated":4053061632,"data_stored":4818592441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1669604822},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,0,6,2,8,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:16:48 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.47799999999999998,"15min":0.46500000000000002},"min":{"1min":0.193,"5min":0.186,"15min":0.17199999999999999},"max":{"1min":0.93400000000000005,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.56000000000000005,"15min":0.50800000000000001},"min":{"1min":0.21099999999999999,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":3.9870000000000001,"5min":6.1520000000000001,"15min":6.5890000000000004},"last":0.23300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:16:48 2023","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":0.71799999999999997,"15min":0.62},"min":{"1min":0.21099999999999999,"5min":0.20999999999999999,"15min":0.188},"max":{"1min":16.763999999999999,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.247},{"interface":"front","average":{"1min":1.3260000000000001,"5min":0.58599999999999997,"15min":0.5},"min":{"1min":0.25,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":16.748000000000001,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.29999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4052140032,"data_stored":4818325504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4070666240,"data_stored":4838494227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8121540608,"data_stored":9655554067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:17:36.598 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:36.598 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:36.599 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:36.599 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:36.600 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:17:56.601 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:17:57.097 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:17:57.098 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:17:57.126 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2288,"stamp":"2023-10-14T01:17:56.433434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29072625484,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2014047,"num_read_kb":7862538,"num_write":1991664,"num_write_kb":7965667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87640,"ondisk_log_size":87640,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21379432,"kb_used_data":15927632,"kb_used_omap":2282,"kb_used_meta":5449493,"kb_avail":261736088,"statfs":{"total":289910292480,"available":268017754112,"internally_reserved":0,"allocated":16309895168,"data_stored":19376006214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":5580281716},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,8,6,13,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23531520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5575,"num_read_kb":22264,"num_write":5808,"num_write_kb":23232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002051"},"pg_stats":[{"pgid":"2.7","version":"92'10322","reported_seq":14269,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:54.551289+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:17:54.551289+0000","last_peered":"2023-10-14T01:17:54.551289+0000","last_clean":"2023-10-14T01:17:54.551289+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:17:54.551289+0000","last_undegraded":"2023-10-14T01:17:54.551289+0000","last_fullsized":"2023-10-14T01:17:54.551289+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6949","reported_seq":10535,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:54.526090+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:17:54.526090+0000","last_peered":"2023-10-14T01:17:54.526090+0000","last_clean":"2023-10-14T01:17:54.526090+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:17:54.526090+0000","last_undegraded":"2023-10-14T01:17:54.526090+0000","last_fullsized":"2023-10-14T01:17:54.526090+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":177,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:16:22.174754+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:16:22.174754+0000","last_peered":"2023-10-14T01:16:22.174754+0000","last_clean":"2023-10-14T01:16:22.174754+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:16:22.174754+0000","last_undegraded":"2023-10-14T01:16:22.174754+0000","last_fullsized":"2023-10-14T01:16:22.174754+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'13722","reported_seq":19351,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:54.536129+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:17:54.536129+0000","last_peered":"2023-10-14T01:17:54.536129+0000","last_clean":"2023-10-14T01:17:54.536129+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:17:54.536129+0000","last_undegraded":"2023-10-14T01:17:54.536129+0000","last_fullsized":"2023-10-14T01:17:54.536129+0000","mapping_epoch":12,"log_start":"75'3700","ondisk_log_start":"75'3700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'495953","reported_seq":984634,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:54.877703+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:17:54.877703+0000","last_peered":"2023-10-14T01:17:54.877703+0000","last_clean":"2023-10-14T01:17:54.877703+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:17:54.877703+0000","last_undegraded":"2023-10-14T01:17:54.877703+0000","last_fullsized":"2023-10-14T01:17:54.877703+0000","mapping_epoch":15,"log_start":"90'485933","ondisk_log_start":"90'485933","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3244,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 451s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7162122240,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":490745,"num_read_kb":1932652,"num_write":491152,"num_write_kb":1964608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'10184","reported_seq":14018,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:53.317406+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:17:53.317406+0000","last_peered":"2023-10-14T01:17:53.317406+0000","last_clean":"2023-10-14T01:17:53.317406+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:17:53.317406+0000","last_undegraded":"2023-10-14T01:17:53.317406+0000","last_fullsized":"2023-10-14T01:17:53.317406+0000","mapping_epoch":12,"log_start":"39'100","ondisk_log_start":"39'100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'503383","reported_seq":999444,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:55.634354+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:17:55.634354+0000","last_peered":"2023-10-14T01:17:55.634354+0000","last_clean":"2023-10-14T01:17:55.634354+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:17:55.634354+0000","last_undegraded":"2023-10-14T01:17:55.634354+0000","last_fullsized":"2023-10-14T01:17:55.634354+0000","mapping_epoch":15,"log_start":"90'493327","ondisk_log_start":"90'493327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2956,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 447s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7293444096,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":498190,"num_read_kb":1961704,"num_write":498747,"num_write_kb":1994988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3946,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:51.262765+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:17:51.262765+0000","last_peered":"2023-10-14T01:17:51.262765+0000","last_clean":"2023-10-14T01:17:51.262765+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:17:51.262765+0000","last_undegraded":"2023-10-14T01:17:51.262765+0000","last_fullsized":"2023-10-14T01:17:51.262765+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'506793","reported_seq":1006997,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:17:54.876091+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:17:54.876091+0000","last_peered":"2023-10-14T01:17:54.876091+0000","last_clean":"2023-10-14T01:17:54.876091+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:17:54.876091+0000","last_undegraded":"2023-10-14T01:17:54.876091+0000","last_fullsized":"2023-10-14T01:17:54.876091+0000","mapping_epoch":15,"log_start":"90'496727","ondisk_log_start":"90'496727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3097,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 455s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7318609920,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":502341,"num_read_kb":1978876,"num_write":502051,"num_write_kb":2008204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1432,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:46.134018+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:17:46.134018+0000","last_peered":"2023-10-14T01:17:46.134018+0000","last_clean":"2023-10-14T01:17:46.134018+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:17:46.134018+0000","last_undegraded":"2023-10-14T01:17:46.134018+0000","last_fullsized":"2023-10-14T01:17:46.134018+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'501476","reported_seq":999537,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:55.620429+0000","last_change":"2023-10-14T01:17:51.285602+0000","last_active":"2023-10-14T01:17:55.620429+0000","last_peered":"2023-10-14T01:17:55.620429+0000","last_clean":"2023-10-14T01:17:55.620429+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:17:55.620429+0000","last_undegraded":"2023-10-14T01:17:55.620429+0000","last_fullsized":"2023-10-14T01:17:55.620429+0000","mapping_epoch":15,"log_start":"90'491433","ondisk_log_start":"90'491433","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7297855507,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":500334,"num_read_kb":1970392,"num_write":498904,"num_write_kb":1995610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'13637","reported_seq":18616,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:54.567027+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:17:54.567027+0000","last_peered":"2023-10-14T01:17:54.567027+0000","last_clean":"2023-10-14T01:17:54.567027+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:17:54.567027+0000","last_undegraded":"2023-10-14T01:17:54.567027+0000","last_fullsized":"2023-10-14T01:17:54.567027+0000","mapping_epoch":12,"log_start":"75'3600","ondisk_log_start":"75'3600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.378816+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:17:10.378816+0000","last_peered":"2023-10-14T01:17:10.378816+0000","last_clean":"2023-10-14T01:17:10.378816+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:17:10.378816+0000","last_undegraded":"2023-10-14T01:17:10.378816+0000","last_fullsized":"2023-10-14T01:17:10.378816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29072031763,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1991610,"num_read_kb":7843624,"num_write":1990854,"num_write_kb":7963410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16307830784,"data_stored":19375857702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40185,"ondisk_log_size":40185,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47356,"ondisk_log_size":47356,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739251,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9955864,"kb_used_data":7962128,"kb_used_omap":1268,"kb_used_meta":1992459,"kb_avail":84415976,"statfs":{"total":96636764160,"available":86441959424,"internally_reserved":0,"allocated":8153219072,"data_stored":9686719239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2040278346},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.314,"15min":0.33300000000000002},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":1.155,"5min":1.155,"15min":2.5139999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":0.36399999999999999,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.44600000000000001,"15min":0.46100000000000002},"min":{"1min":0.24099999999999999,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":3.883,"5min":3.883,"15min":4.1840000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.49099999999999999,"15min":0.57699999999999996},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":3.915,"5min":3.915,"15min":16.747},"last":0.47999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739252,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5754148,"kb_used_data":3991836,"kb_used_omap":510,"kb_used_meta":1761793,"kb_avail":88617692,"statfs":{"total":96636764160,"available":90744516608,"internally_reserved":0,"allocated":4087640064,"data_stored":4854720134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1804076116},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.222,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":7.2809999999999997,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51900000000000002,"15min":0.48999999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":3.2440000000000002,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":1.5289999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:17:09 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.40000000000000002,"15min":0.39500000000000002},"min":{"1min":0.23400000000000001,"5min":0.20999999999999999,"15min":0.193},"max":{"1min":0.54800000000000004,"5min":3.7559999999999998,"15min":5.6040000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.442},"min":{"1min":0.28399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.57499999999999996,"5min":3.7839999999999998,"15min":5.8529999999999998},"last":0.42999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739254,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5669420,"kb_used_data":3973668,"kb_used_omap":502,"kb_used_meta":1695241,"kb_avail":88702420,"statfs":{"total":96636764160,"available":90831278080,"internally_reserved":0,"allocated":4069036032,"data_stored":4834566841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1735927254},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.48099999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.5059999999999998,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.64500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":6.8369999999999997,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:17:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.63300000000000001},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.7610000000000001,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.58099999999999996,"15min":0.497},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.58699999999999997,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4068175872,"data_stored":4834361344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4086378496,"data_stored":4854206483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8153276416,"data_stored":9687289875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:17:57.128 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:57.128 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:57.128 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:57.129 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:17:57.129 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:09.975 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:18:09.971+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:18:09.989 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:18:09.975+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:18:09.992 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:18:09.991+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:18:17.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:17.715 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:18:17.715 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:18:17.744 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2298,"stamp":"2023-10-14T01:18:16.436669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29103169356,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2021729,"num_read_kb":7893182,"num_write":1999214,"num_write_kb":7995867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87805,"ondisk_log_size":87805,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21568276,"kb_used_data":15987772,"kb_used_omap":2282,"kb_used_meta":5578197,"kb_avail":261547244,"statfs":{"total":289910292480,"available":267824377856,"internally_reserved":0,"allocated":16371478528,"data_stored":19437589574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":5712074612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,4,7,9,8,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22548480,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5730,"num_read_kb":22864,"num_write":5580,"num_write_kb":22320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001875"},"pg_stats":[{"pgid":"2.7","version":"92'10362","reported_seq":14321,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:13.769217+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:18:13.769217+0000","last_peered":"2023-10-14T01:18:13.769217+0000","last_clean":"2023-10-14T01:18:13.769217+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:18:13.769217+0000","last_undegraded":"2023-10-14T01:18:13.769217+0000","last_fullsized":"2023-10-14T01:18:13.769217+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6976","reported_seq":10570,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:13.761210+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:18:13.761210+0000","last_peered":"2023-10-14T01:18:13.761210+0000","last_clean":"2023-10-14T01:18:13.761210+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:18:13.761210+0000","last_undegraded":"2023-10-14T01:18:13.761210+0000","last_fullsized":"2023-10-14T01:18:13.761210+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6976,"log_dups_size":0,"ondisk_log_size":6976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":178,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:32.190082+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:17:32.190082+0000","last_peered":"2023-10-14T01:17:32.190082+0000","last_clean":"2023-10-14T01:17:32.190082+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:17:32.190082+0000","last_undegraded":"2023-10-14T01:17:32.190082+0000","last_fullsized":"2023-10-14T01:17:32.190082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'13775","reported_seq":19421,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:13.778994+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:18:13.778994+0000","last_peered":"2023-10-14T01:18:13.778994+0000","last_clean":"2023-10-14T01:18:13.778994+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:18:13.778994+0000","last_undegraded":"2023-10-14T01:18:13.778994+0000","last_fullsized":"2023-10-14T01:18:13.778994+0000","mapping_epoch":12,"log_start":"75'3700","ondisk_log_start":"75'3700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'497886","reported_seq":988438,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:15.037312+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:18:15.037312+0000","last_peered":"2023-10-14T01:18:15.037312+0000","last_clean":"2023-10-14T01:18:15.037312+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:18:15.037312+0000","last_undegraded":"2023-10-14T01:18:15.037312+0000","last_fullsized":"2023-10-14T01:18:15.037312+0000","mapping_epoch":15,"log_start":"91'487833","ondisk_log_start":"91'487833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3412,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 471s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7169445888,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":492605,"num_read_kb":1940084,"num_write":492967,"num_write_kb":1971868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'10238","reported_seq":14085,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:14.629122+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:18:14.629122+0000","last_peered":"2023-10-14T01:18:14.629122+0000","last_clean":"2023-10-14T01:18:14.629122+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:18:14.629122+0000","last_undegraded":"2023-10-14T01:18:14.629122+0000","last_fullsized":"2023-10-14T01:18:14.629122+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'505391","reported_seq":1003368,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:15.753143+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:18:15.753143+0000","last_peered":"2023-10-14T01:18:15.753143+0000","last_clean":"2023-10-14T01:18:15.753143+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:18:15.753143+0000","last_undegraded":"2023-10-14T01:18:15.753143+0000","last_fullsized":"2023-10-14T01:18:15.753143+0000","mapping_epoch":15,"log_start":"91'495328","ondisk_log_start":"91'495328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3070,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 467s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7301283840,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":500094,"num_read_kb":1969304,"num_write":500673,"num_write_kb":2002692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3958,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:11.737273+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:18:11.737273+0000","last_peered":"2023-10-14T01:18:11.737273+0000","last_clean":"2023-10-14T01:18:11.737273+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:18:11.737273+0000","last_undegraded":"2023-10-14T01:18:11.737273+0000","last_fullsized":"2023-10-14T01:18:11.737273+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'508790","reported_seq":1010944,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:15.009363+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:18:15.009363+0000","last_peered":"2023-10-14T01:18:15.009363+0000","last_clean":"2023-10-14T01:18:15.009363+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:18:15.009363+0000","last_undegraded":"2023-10-14T01:18:15.009363+0000","last_fullsized":"2023-10-14T01:18:15.009363+0000","mapping_epoch":15,"log_start":"91'498727","ondisk_log_start":"91'498727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3223,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 475s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7326326784,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":504277,"num_read_kb":1986580,"num_write":503959,"num_write_kb":2015836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1436,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:06.634185+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:18:06.634185+0000","last_peered":"2023-10-14T01:18:06.634185+0000","last_clean":"2023-10-14T01:18:06.634185+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:18:06.634185+0000","last_undegraded":"2023-10-14T01:18:06.634185+0000","last_fullsized":"2023-10-14T01:18:06.634185+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'503377","reported_seq":1003428,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:15.754779+0000","last_change":"2023-10-14T01:18:13.031984+0000","last_active":"2023-10-14T01:18:15.754779+0000","last_peered":"2023-10-14T01:18:15.754779+0000","last_clean":"2023-10-14T01:18:15.754779+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:18:15.754779+0000","last_undegraded":"2023-10-14T01:18:15.754779+0000","last_fullsized":"2023-10-14T01:18:15.754779+0000","mapping_epoch":15,"log_start":"91'493333","ondisk_log_start":"91'493333","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7305519123,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":502316,"num_read_kb":1978300,"num_write":500805,"num_write_kb":2003214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'13690","reported_seq":18686,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:13.796410+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:18:13.796410+0000","last_peered":"2023-10-14T01:18:13.796410+0000","last_clean":"2023-10-14T01:18:13.796410+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:18:13.796410+0000","last_undegraded":"2023-10-14T01:18:13.796410+0000","last_fullsized":"2023-10-14T01:18:13.796410+0000","mapping_epoch":12,"log_start":"75'3600","ondisk_log_start":"75'3600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.378816+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:17:10.378816+0000","last_peered":"2023-10-14T01:17:10.378816+0000","last_clean":"2023-10-14T01:17:10.378816+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:17:10.378816+0000","last_undegraded":"2023-10-14T01:17:10.378816+0000","last_fullsized":"2023-10-14T01:17:10.378816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29102575635,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":1999292,"num_read_kb":7874268,"num_write":1998404,"num_write_kb":7993610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16369741824,"data_stored":19437768742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40223,"ondisk_log_size":40223,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47483,"ondisk_log_size":47483,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739255,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10073632,"kb_used_data":7992152,"kb_used_omap":1268,"kb_used_meta":2080203,"kb_avail":84298208,"statfs":{"total":96636764160,"available":86321364992,"internally_reserved":0,"allocated":8183963648,"data_stored":9717463815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2130128202},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.314,"15min":0.33300000000000002},"min":{"1min":0.193,"5min":0.17000000000000001,"15min":0.16700000000000001},"max":{"1min":1.155,"5min":1.155,"15min":2.5139999999999998},"last":0.622},{"interface":"front","average":{"1min":0.26700000000000002,"5min":0.35499999999999998,"15min":0.36399999999999999},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17199999999999999},"max":{"1min":0.36399999999999999,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:17:31 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.44600000000000001,"15min":0.46100000000000002},"min":{"1min":0.24099999999999999,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":3.883,"5min":3.883,"15min":4.1840000000000002},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.49099999999999999,"15min":0.57699999999999996},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.18099999999999999},"max":{"1min":3.915,"5min":3.915,"15min":16.747},"last":0.41099999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739256,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5774284,"kb_used_data":4007108,"kb_used_omap":510,"kb_used_meta":1766657,"kb_avail":88597556,"statfs":{"total":96636764160,"available":90723897344,"internally_reserved":0,"allocated":4103278592,"data_stored":4870358662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1809056852},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,3,7,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.56399999999999995,"15min":0.495},"min":{"1min":0.25800000000000001,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":5.5499999999999998,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.57199999999999995,"15min":0.503},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":4.9180000000000001,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.49299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.193},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.6040000000000001},"last":0.373},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.54100000000000004,"15min":0.46999999999999997},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":5.617,"5min":5.617,"15min":5.8529999999999998},"last":0.40000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739258,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5720360,"kb_used_data":3988512,"kb_used_omap":502,"kb_used_meta":1731337,"kb_avail":88651480,"statfs":{"total":96636764160,"available":90779115520,"internally_reserved":0,"allocated":4084236288,"data_stored":4849767097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1772889558},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,6,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.48099999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.5059999999999998,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.64500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":6.8369999999999997,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.32900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:17:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.63300000000000001},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.7610000000000001,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.58099999999999996,"15min":0.497},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.58699999999999997,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4083412992,"data_stored":4849598464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4102049792,"data_stored":4869877779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8184279040,"data_stored":9718292499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:18:17.745 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:17.746 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:17.746 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:17.747 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:17.747 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:37.748 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:38.297 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:18:38.298 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:18:38.332 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2308,"stamp":"2023-10-14T01:18:36.440007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29134270284,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2029614,"num_read_kb":7924618,"num_write":2006878,"num_write_kb":8026523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87688,"ondisk_log_size":87688,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21805612,"kb_used_data":16048788,"kb_used_omap":2282,"kb_used_meta":5754517,"kb_avail":261309908,"statfs":{"total":289910292480,"available":267581345792,"internally_reserved":0,"allocated":16433958912,"data_stored":19500069958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":5892626292},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,6,14,19,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23367680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5941,"num_read_kb":23684,"num_write":5763,"num_write_kb":23052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001925"},"pg_stats":[{"pgid":"2.7","version":"92'10383","reported_seq":14354,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.760749+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:18:34.760749+0000","last_peered":"2023-10-14T01:18:34.760749+0000","last_clean":"2023-10-14T01:18:34.760749+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:18:34.760749+0000","last_undegraded":"2023-10-14T01:18:34.760749+0000","last_fullsized":"2023-10-14T01:18:34.760749+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"92'6990","reported_seq":10592,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.751611+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:18:34.751611+0000","last_peered":"2023-10-14T01:18:34.751611+0000","last_clean":"2023-10-14T01:18:34.751611+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:18:34.751611+0000","last_undegraded":"2023-10-14T01:18:34.751611+0000","last_fullsized":"2023-10-14T01:18:34.751611+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":6990,"log_dups_size":0,"ondisk_log_size":6990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":178,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:32.190082+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:17:32.190082+0000","last_peered":"2023-10-14T01:17:32.190082+0000","last_clean":"2023-10-14T01:17:32.190082+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:17:32.190082+0000","last_undegraded":"2023-10-14T01:17:32.190082+0000","last_fullsized":"2023-10-14T01:17:32.190082+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"92'13803","reported_seq":19465,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.767223+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:18:34.767223+0000","last_peered":"2023-10-14T01:18:34.767223+0000","last_clean":"2023-10-14T01:18:34.767223+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:18:34.767223+0000","last_undegraded":"2023-10-14T01:18:34.767223+0000","last_fullsized":"2023-10-14T01:18:34.767223+0000","mapping_epoch":12,"log_start":"77'3800","ondisk_log_start":"77'3800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"92'499901","reported_seq":992447,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:35.114435+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:18:35.114435+0000","last_peered":"2023-10-14T01:18:35.114435+0000","last_clean":"2023-10-14T01:18:35.114435+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:18:35.114435+0000","last_undegraded":"2023-10-14T01:18:35.114435+0000","last_fullsized":"2023-10-14T01:18:35.114435+0000","mapping_epoch":15,"log_start":"91'489833","ondisk_log_start":"91'489833","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3550,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 491s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7177269248,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":494583,"num_read_kb":1947964,"num_write":494889,"num_write_kb":1979556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"92'10253","reported_seq":14112,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:35.105546+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:18:35.105546+0000","last_peered":"2023-10-14T01:18:35.105546+0000","last_clean":"2023-10-14T01:18:35.105546+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:18:35.105546+0000","last_undegraded":"2023-10-14T01:18:35.105546+0000","last_fullsized":"2023-10-14T01:18:35.105546+0000","mapping_epoch":12,"log_start":"41'200","ondisk_log_start":"41'200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"92'507434","reported_seq":1007391,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:35.849815+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:18:35.849815+0000","last_peered":"2023-10-14T01:18:35.849815+0000","last_clean":"2023-10-14T01:18:35.849815+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:18:35.849815+0000","last_undegraded":"2023-10-14T01:18:35.849815+0000","last_fullsized":"2023-10-14T01:18:35.849815+0000","mapping_epoch":15,"log_start":"91'497428","ondisk_log_start":"91'497428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3208,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 488s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7309234176,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":502062,"num_read_kb":1977152,"num_write":502620,"num_write_kb":2010480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3970,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:32.210071+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:18:32.210071+0000","last_peered":"2023-10-14T01:18:32.210071+0000","last_clean":"2023-10-14T01:18:32.210071+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:18:32.210071+0000","last_undegraded":"2023-10-14T01:18:32.210071+0000","last_fullsized":"2023-10-14T01:18:32.210071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"92'510822","reported_seq":1014976,"reported_epoch":92,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:35.136317+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:18:35.136317+0000","last_peered":"2023-10-14T01:18:35.136317+0000","last_clean":"2023-10-14T01:18:35.136317+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:18:35.136317+0000","last_undegraded":"2023-10-14T01:18:35.136317+0000","last_fullsized":"2023-10-14T01:18:35.136317+0000","mapping_epoch":15,"log_start":"91'500727","ondisk_log_start":"91'500727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3403,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 495s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7334027264,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":506262,"num_read_kb":1994500,"num_write":505867,"num_write_kb":2023468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1440,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:27.207476+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:18:27.207476+0000","last_peered":"2023-10-14T01:18:27.207476+0000","last_clean":"2023-10-14T01:18:27.207476+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:18:27.207476+0000","last_undegraded":"2023-10-14T01:18:27.207476+0000","last_fullsized":"2023-10-14T01:18:27.207476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"92'505264","reported_seq":1007274,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:35.857602+0000","last_change":"2023-10-14T01:18:34.991868+0000","last_active":"2023-10-14T01:18:35.857602+0000","last_peered":"2023-10-14T01:18:35.857602+0000","last_clean":"2023-10-14T01:18:35.857602+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:18:35.857602+0000","last_undegraded":"2023-10-14T01:18:35.857602+0000","last_fullsized":"2023-10-14T01:18:35.857602+0000","mapping_epoch":15,"log_start":"91'495233","ondisk_log_start":"91'495233","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7313145875,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":504270,"num_read_kb":1986088,"num_write":502692,"num_write_kb":2010762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"92'13718","reported_seq":18730,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:34.817003+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:18:34.817003+0000","last_peered":"2023-10-14T01:18:34.817003+0000","last_clean":"2023-10-14T01:18:34.817003+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:18:34.817003+0000","last_undegraded":"2023-10-14T01:18:34.817003+0000","last_fullsized":"2023-10-14T01:18:34.817003+0000","mapping_epoch":12,"log_start":"75'3700","ondisk_log_start":"75'3700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":266,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-14T01:17:10.378816+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:17:10.378816+0000","last_peered":"2023-10-14T01:17:10.378816+0000","last_clean":"2023-10-14T01:17:10.378816+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:17:10.378816+0000","last_undegraded":"2023-10-14T01:17:10.378816+0000","last_fullsized":"2023-10-14T01:17:10.378816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29133676563,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2007177,"num_read_kb":7905704,"num_write":2006068,"num_write_kb":8024266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16432488448,"data_stored":19500515366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40200,"ondisk_log_size":40200,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47389,"ondisk_log_size":47389,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739259,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10194420,"kb_used_data":8022700,"kb_used_omap":1268,"kb_used_meta":2170443,"kb_avail":84177420,"statfs":{"total":96636764160,"available":86197678080,"internally_reserved":0,"allocated":8215244800,"data_stored":9748744967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2222533962},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16700000000000001},"max":{"1min":0.83899999999999997,"5min":1.155,"15min":2.5139999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.379,"15min":0.36399999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.7170000000000001,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":0.61899999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.497,"15min":0.57499999999999996},"min":{"1min":0.27700000000000002,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":0.64900000000000002,"5min":3.915,"15min":16.747},"last":0.40999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739260,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5821404,"kb_used_data":4022292,"kb_used_omap":510,"kb_used_meta":1798593,"kb_avail":88550436,"statfs":{"total":96636764160,"available":90675646464,"internally_reserved":0,"allocated":4118827008,"data_stored":4885907078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1841759316},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1,4,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.56399999999999995,"15min":0.495},"min":{"1min":0.25800000000000001,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":5.5499999999999998,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.57199999999999995,"15min":0.503},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":4.9180000000000001,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":5.3360000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.49299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.193},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.6040000000000001},"last":0.249},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.54100000000000004,"15min":0.46999999999999997},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":5.617,"5min":5.617,"15min":5.8529999999999998},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359739262,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5789788,"kb_used_data":4003796,"kb_used_omap":502,"kb_used_meta":1785481,"kb_avail":88582052,"statfs":{"total":96636764160,"available":90708021248,"internally_reserved":0,"allocated":4099887104,"data_stored":4865417913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1828333014},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,7,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:17:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.48099999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.5059999999999998,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.311},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.64500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19600000000000001,"5min":0.191,"15min":0.16400000000000001},"max":{"1min":6.8369999999999997,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:17:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.63300000000000001},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":6.7610000000000001,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.377},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.58099999999999996,"15min":0.497},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.58699999999999997,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4099108864,"data_stored":4865294336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4117753856,"data_stored":4885581843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8215625728,"data_stored":9749639187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:18:38.334 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:38.335 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:38.335 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:38.336 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:38.336 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:18:45.205 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:18:45.203+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:18:58.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:18:58.845 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:18:58.846 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:18:58.892 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2320,"stamp":"2023-10-14T01:18:58.443708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29164220236,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2037174,"num_read_kb":7954782,"num_write":2014259,"num_write_kb":8056047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87844,"ondisk_log_size":87844,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22000944,"kb_used_data":16108052,"kb_used_omap":2282,"kb_used_meta":5890581,"kb_avail":261114576,"statfs":{"total":289910292480,"available":267381325824,"internally_reserved":0,"allocated":16494645248,"data_stored":19560743091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":6031955828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,4,6,4,12,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15015936,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3762,"num_read_kb":15004,"num_write":3702,"num_write_kb":14808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001986"},"pg_stats":[{"pgid":"2.7","version":"93'10422","reported_seq":14407,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:55.627483+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:18:55.627483+0000","last_peered":"2023-10-14T01:18:55.627483+0000","last_clean":"2023-10-14T01:18:55.627483+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:18:55.627483+0000","last_undegraded":"2023-10-14T01:18:55.627483+0000","last_fullsized":"2023-10-14T01:18:55.627483+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'7016","reported_seq":10628,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:55.616847+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:18:55.616847+0000","last_peered":"2023-10-14T01:18:55.616847+0000","last_clean":"2023-10-14T01:18:55.616847+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:18:55.616847+0000","last_undegraded":"2023-10-14T01:18:55.616847+0000","last_fullsized":"2023-10-14T01:18:55.616847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":179,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:10.751629+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:18:10.751629+0000","last_peered":"2023-10-14T01:18:10.751629+0000","last_clean":"2023-10-14T01:18:10.751629+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:18:10.751629+0000","last_undegraded":"2023-10-14T01:18:10.751629+0000","last_fullsized":"2023-10-14T01:18:10.751629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'13855","reported_seq":19534,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:55.634280+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:18:55.634280+0000","last_peered":"2023-10-14T01:18:55.634280+0000","last_clean":"2023-10-14T01:18:55.634280+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:18:55.634280+0000","last_undegraded":"2023-10-14T01:18:55.634280+0000","last_fullsized":"2023-10-14T01:18:55.634280+0000","mapping_epoch":12,"log_start":"77'3800","ondisk_log_start":"77'3800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'501817","reported_seq":996247,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:55.211677+0000","last_change":"2023-10-14T01:10:23.165253+0000","last_active":"2023-10-14T01:18:55.211677+0000","last_peered":"2023-10-14T01:18:55.211677+0000","last_clean":"2023-10-14T01:18:55.211677+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:18:55.211677+0000","last_undegraded":"2023-10-14T01:18:55.211677+0000","last_fullsized":"2023-10-14T01:18:55.211677+0000","mapping_epoch":15,"log_start":"91'491734","ondisk_log_start":"91'491734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3730,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 512s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7184539648,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":496453,"num_read_kb":1955432,"num_write":496681,"num_write_kb":1986724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'10301","reported_seq":14173,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:54.102644+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:18:54.102644+0000","last_peered":"2023-10-14T01:18:54.102644+0000","last_clean":"2023-10-14T01:18:54.102644+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:18:54.102644+0000","last_undegraded":"2023-10-14T01:18:54.102644+0000","last_fullsized":"2023-10-14T01:18:54.102644+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'509390","reported_seq":1011290,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:55.910993+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:18:55.910993+0000","last_peered":"2023-10-14T01:18:55.910993+0000","last_clean":"2023-10-14T01:18:55.910993+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:18:55.910993+0000","last_undegraded":"2023-10-14T01:18:55.910993+0000","last_fullsized":"2023-10-14T01:18:55.910993+0000","mapping_epoch":15,"log_start":"91'499328","ondisk_log_start":"91'499328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3394,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 508s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7316639744,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":503989,"num_read_kb":1984844,"num_write":504444,"num_write_kb":2017776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3983,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:52.727329+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:18:52.727329+0000","last_peered":"2023-10-14T01:18:52.727329+0000","last_clean":"2023-10-14T01:18:52.727329+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:18:52.727329+0000","last_undegraded":"2023-10-14T01:18:52.727329+0000","last_fullsized":"2023-10-14T01:18:52.727329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'512826","reported_seq":1018905,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:18:55.224010+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:18:55.224010+0000","last_peered":"2023-10-14T01:18:55.224010+0000","last_clean":"2023-10-14T01:18:55.224010+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:18:55.224010+0000","last_undegraded":"2023-10-14T01:18:55.224010+0000","last_fullsized":"2023-10-14T01:18:55.224010+0000","mapping_epoch":15,"log_start":"91'502727","ondisk_log_start":"91'502727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3553,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 516s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7341752320,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":508175,"num_read_kb":2002124,"num_write":507768,"num_write_kb":2031072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1445,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:47.569545+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:18:47.569545+0000","last_peered":"2023-10-14T01:18:47.569545+0000","last_clean":"2023-10-14T01:18:47.569545+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:18:47.569545+0000","last_undegraded":"2023-10-14T01:18:47.569545+0000","last_fullsized":"2023-10-14T01:18:47.569545+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'507128","reported_seq":1010996,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:55.926895+0000","last_change":"2023-10-14T01:18:54.146207+0000","last_active":"2023-10-14T01:18:55.926895+0000","last_peered":"2023-10-14T01:18:55.926895+0000","last_clean":"2023-10-14T01:18:55.926895+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:18:55.926895+0000","last_undegraded":"2023-10-14T01:18:55.926895+0000","last_fullsized":"2023-10-14T01:18:55.926895+0000","mapping_epoch":15,"log_start":"91'497033","ondisk_log_start":"91'497033","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7320694803,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":506120,"num_read_kb":1993468,"num_write":504556,"num_write_kb":2018218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'13770","reported_seq":18799,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:55.676031+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:18:55.676031+0000","last_peered":"2023-10-14T01:18:55.676031+0000","last_clean":"2023-10-14T01:18:55.676031+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:18:55.676031+0000","last_undegraded":"2023-10-14T01:18:55.676031+0000","last_fullsized":"2023-10-14T01:18:55.676031+0000","mapping_epoch":12,"log_start":"75'3700","ondisk_log_start":"75'3700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":267,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:17:32.248721+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:17:32.248721+0000","last_peered":"2023-10-14T01:17:32.248721+0000","last_clean":"2023-10-14T01:17:32.248721+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:17:32.248721+0000","last_undegraded":"2023-10-14T01:17:32.248721+0000","last_fullsized":"2023-10-14T01:17:32.248721+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29163626515,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2014737,"num_read_kb":7935868,"num_write":2013449,"num_write_kb":8053790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16493125632,"data_stored":19561152550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40339,"ondisk_log_size":40339,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47406,"ondisk_log_size":47406,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739263,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10292092,"kb_used_data":8052336,"kb_used_omap":1268,"kb_used_meta":2238475,"kb_avail":84079748,"statfs":{"total":96636764160,"available":86097661952,"internally_reserved":0,"allocated":8245592064,"data_stored":9779087830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2292198730},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16700000000000001},"max":{"1min":0.83899999999999997,"5min":1.155,"15min":2.5139999999999998},"last":0.38},{"interface":"front","average":{"1min":0.374,"5min":0.379,"15min":0.36399999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.7170000000000001,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":0.61899999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.497,"15min":0.57499999999999996},"min":{"1min":0.27700000000000002,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":0.64900000000000002,"5min":3.915,"15min":16.747},"last":0.41299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739264,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5872256,"kb_used_data":4037048,"kb_used_omap":510,"kb_used_meta":1834689,"kb_avail":88499584,"statfs":{"total":96636764160,"available":90623574016,"internally_reserved":0,"allocated":4133937152,"data_stored":4901012821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1878721620},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.56399999999999995,"15min":0.495},"min":{"1min":0.25800000000000001,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":5.5499999999999998,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.57199999999999995,"15min":0.503},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":4.9180000000000001,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:18:13 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.49299999999999999,"15min":0.42299999999999999},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.193},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.6040000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.54100000000000004,"15min":0.46999999999999997},"min":{"1min":0.29699999999999999,"5min":0.249,"15min":0.249},"max":{"1min":5.617,"5min":5.617,"15min":5.8529999999999998},"last":0.40000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739266,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5836596,"kb_used_data":4018668,"kb_used_omap":502,"kb_used_meta":1817417,"kb_avail":88535244,"statfs":{"total":96636764160,"available":90660089856,"internally_reserved":0,"allocated":4115116032,"data_stored":4880642440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1861035478},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,4,12,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:52 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.47899999999999998,"15min":0.48099999999999998},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39600000000000002,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.373},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.65700000000000003,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.17399999999999999},"max":{"1min":2.4169999999999998,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.27900000000000003}]},{"osd":2,"last update":"Sat Oct 14 01:18:52 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.68000000000000005,"15min":0.63700000000000001},"min":{"1min":0.20200000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.68700000000000006,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.313},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.53900000000000003,"15min":0.497},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49199999999999999,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4114227200,"data_stored":4880412672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4132864000,"data_stored":4900691987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8246034432,"data_stored":9780047891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:18:58.894 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:58.894 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:58.895 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:58.895 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:18:58.896 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.3 2023-10-14T01:18:59.393 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.3 on osd.0 to deep-scrub 2023-10-14T01:18:59.428 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.0 2023-10-14T01:18:59.930 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2023-10-14T01:18:59.975 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1 2023-10-14T01:19:00.509 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.1 on osd.0 to deep-scrub 2023-10-14T01:19:00.540 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:19:01.101 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:19:01.131 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:21.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:21.650 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:19:21.650 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:19:21.672 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2331,"stamp":"2023-10-14T01:19:20.447355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29198061388,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2045818,"num_read_kb":7989206,"num_write":2022568,"num_write_kb":8089283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87901,"ondisk_log_size":87901,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21783876,"kb_used_data":16179432,"kb_used_omap":2282,"kb_used_meta":5602133,"kb_avail":261331644,"statfs":{"total":289910292480,"available":267603603456,"internally_reserved":0,"allocated":16567738368,"data_stored":19633836211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":5736585076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,6,6,10,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19038208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4848,"num_read_kb":19300,"num_write":4675,"num_write_kb":18700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001951"},"pg_stats":[{"pgid":"2.7","version":"93'10455","reported_seq":14452,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.949664+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:19:14.949664+0000","last_peered":"2023-10-14T01:19:14.949664+0000","last_clean":"2023-10-14T01:19:14.949664+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:19:14.949664+0000","last_undegraded":"2023-10-14T01:19:14.949664+0000","last_fullsized":"2023-10-14T01:19:14.949664+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'7038","reported_seq":10658,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.915592+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:19:14.915592+0000","last_peered":"2023-10-14T01:19:14.915592+0000","last_clean":"2023-10-14T01:19:14.915592+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:19:14.915592+0000","last_undegraded":"2023-10-14T01:19:14.915592+0000","last_fullsized":"2023-10-14T01:19:14.915592+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7038,"log_dups_size":0,"ondisk_log_size":7038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":179,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:10.751629+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:18:10.751629+0000","last_peered":"2023-10-14T01:18:10.751629+0000","last_clean":"2023-10-14T01:18:10.751629+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:18:10.751629+0000","last_undegraded":"2023-10-14T01:18:10.751629+0000","last_fullsized":"2023-10-14T01:18:10.751629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'13899","reported_seq":19594,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.952871+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:19:14.952871+0000","last_peered":"2023-10-14T01:19:14.952871+0000","last_clean":"2023-10-14T01:19:14.952871+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:19:14.952871+0000","last_undegraded":"2023-10-14T01:19:14.952871+0000","last_fullsized":"2023-10-14T01:19:14.952871+0000","mapping_epoch":12,"log_start":"77'3800","ondisk_log_start":"77'3800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'504168","reported_seq":1001007,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:20.350366+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:19:20.350366+0000","last_peered":"2023-10-14T01:19:20.350366+0000","last_clean":"2023-10-14T01:19:20.350366+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:19:20.350366+0000","last_undegraded":"2023-10-14T01:19:20.350366+0000","last_fullsized":"2023-10-14T01:19:20.350366+0000","mapping_epoch":15,"log_start":"92'494134","ondisk_log_start":"92'494134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7193944064,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":498849,"num_read_kb":1964960,"num_write":498995,"num_write_kb":1995980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'10379","reported_seq":14266,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:18.803333+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:19:18.803333+0000","last_peered":"2023-10-14T01:19:18.803333+0000","last_clean":"2023-10-14T01:19:18.803333+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:19:18.803333+0000","last_undegraded":"2023-10-14T01:19:18.803333+0000","last_fullsized":"2023-10-14T01:19:18.803333+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'511321","reported_seq":1015225,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:16.037917+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:19:16.037917+0000","last_peered":"2023-10-14T01:19:16.037917+0000","last_clean":"2023-10-14T01:19:16.037917+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:19:16.037917+0000","last_undegraded":"2023-10-14T01:19:16.037917+0000","last_fullsized":"2023-10-14T01:19:16.037917+0000","mapping_epoch":15,"log_start":"92'501228","ondisk_log_start":"92'501228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3553,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 528s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7324106752,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":505977,"num_read_kb":1992768,"num_write":506264,"num_write_kb":2025056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":3998,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:18.294487+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:19:18.294487+0000","last_peered":"2023-10-14T01:19:18.294487+0000","last_clean":"2023-10-14T01:19:18.294487+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:19:18.294487+0000","last_undegraded":"2023-10-14T01:19:18.294487+0000","last_fullsized":"2023-10-14T01:19:18.294487+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'515304","reported_seq":1023771,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:20.335242+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:19:20.335242+0000","last_peered":"2023-10-14T01:19:20.335242+0000","last_clean":"2023-10-14T01:19:20.335242+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:19:20.335242+0000","last_undegraded":"2023-10-14T01:19:20.335242+0000","last_fullsized":"2023-10-14T01:19:20.335242+0000","mapping_epoch":15,"log_start":"92'505227","ondisk_log_start":"92'505227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3754,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 541s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7351255040,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":510547,"num_read_kb":2011592,"num_write":510102,"num_write_kb":2040408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1451,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.924253+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:19:14.924253+0000","last_peered":"2023-10-14T01:19:14.924253+0000","last_clean":"2023-10-14T01:19:14.924253+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:19:14.924253+0000","last_undegraded":"2023-10-14T01:19:14.924253+0000","last_fullsized":"2023-10-14T01:19:14.924253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'509005","reported_seq":1014767,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:16.037029+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:19:16.037029+0000","last_peered":"2023-10-14T01:19:16.037029+0000","last_clean":"2023-10-14T01:19:16.037029+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:19:16.037029+0000","last_undegraded":"2023-10-14T01:19:16.037029+0000","last_fullsized":"2023-10-14T01:19:16.037029+0000","mapping_epoch":15,"log_start":"91'498934","ondisk_log_start":"91'498934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":52,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 9s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7328161811,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":508008,"num_read_kb":2000972,"num_write":506397,"num_write_kb":2025582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'13814","reported_seq":18859,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:14.946045+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:19:14.946045+0000","last_peered":"2023-10-14T01:19:14.946045+0000","last_clean":"2023-10-14T01:19:14.946045+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:19:14.946045+0000","last_undegraded":"2023-10-14T01:19:14.946045+0000","last_fullsized":"2023-10-14T01:19:14.946045+0000","mapping_epoch":12,"log_start":"77'3800","ondisk_log_start":"77'3800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:42.215993+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:18:42.215993+0000","last_peered":"2023-10-14T01:18:42.215993+0000","last_clean":"2023-10-14T01:18:42.215993+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:18:42.215993+0000","last_undegraded":"2023-10-14T01:18:42.215993+0000","last_fullsized":"2023-10-14T01:18:42.215993+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29197467667,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2023381,"num_read_kb":7970292,"num_write":2021758,"num_write_kb":8087026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16564793344,"data_stored":19632820262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40275,"ondisk_log_size":40275,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47527,"ondisk_log_size":47527,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739268,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9928976,"kb_used_data":8090120,"kb_used_omap":1268,"kb_used_meta":1837579,"kb_avail":84442864,"statfs":{"total":96636764160,"available":86469492736,"internally_reserved":0,"allocated":8284282880,"data_stored":9817778646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":1881681226},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.317,"15min":0.33200000000000002},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16700000000000001},"max":{"1min":0.83899999999999997,"5min":1.155,"15min":2.5139999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.379,"15min":0.36399999999999999},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":1.7170000000000001,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.27300000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:18:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.223,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":0.61899999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.497,"15min":0.57499999999999996},"min":{"1min":0.27700000000000002,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":0.64900000000000002,"5min":3.915,"15min":16.747},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739268,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5940924,"kb_used_data":4051572,"kb_used_omap":510,"kb_used_meta":1888833,"kb_avail":88430916,"statfs":{"total":96636764160,"available":90553257984,"internally_reserved":0,"allocated":4148809728,"data_stored":4915885397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1934165076},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.53400000000000003,"15min":0.48999999999999999},"min":{"1min":0.191,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":1.9410000000000001,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.19600000000000001,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":5.3360000000000003,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.58799999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:19:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.44800000000000001,"15min":0.41299999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.193},"max":{"1min":0.441,"5min":5.5890000000000004,"15min":5.6040000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.495,"15min":0.45800000000000002},"min":{"1min":0.27400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.47399999999999998,"5min":5.617,"15min":5.8529999999999998},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739271,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5913976,"kb_used_data":4037740,"kb_used_omap":502,"kb_used_meta":1875721,"kb_avail":88457864,"statfs":{"total":96636764160,"available":90580852736,"internally_reserved":0,"allocated":4134645760,"data_stored":4900172168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1920738774},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:52 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.47899999999999998,"15min":0.48099999999999998},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39600000000000002,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.65700000000000003,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.17399999999999999},"max":{"1min":2.4169999999999998,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:18:52 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.68000000000000005,"15min":0.63700000000000001},"min":{"1min":0.20200000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.68700000000000006,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.37},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.53900000000000003,"15min":0.497},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49199999999999999,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4133269504,"data_stored":4899454976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4147859456,"data_stored":4915687443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8283664384,"data_stored":9817677843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:19:21.674 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:21.675 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:21.675 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:21.676 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:41.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:19:42.198 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:19:42.199 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:19:42.224 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2341,"stamp":"2023-10-14T01:19:40.450927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29225111372,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2052586,"num_read_kb":8016134,"num_write":2029233,"num_write_kb":8115943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87927,"ondisk_log_size":87927,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22005760,"kb_used_data":16236836,"kb_used_omap":2282,"kb_used_meta":5766613,"kb_avail":261109760,"statfs":{"total":289910292480,"available":267376394240,"internally_reserved":0,"allocated":16626520064,"data_stored":19692617907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2336908,"internal_metadata":5905012596},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,4,20,16,19],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15687680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3944,"num_read_kb":15688,"num_write":3853,"num_write_kb":15412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002088"},"pg_stats":[{"pgid":"2.7","version":"93'10491","reported_seq":14500,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.649718+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:19:33.649718+0000","last_peered":"2023-10-14T01:19:33.649718+0000","last_clean":"2023-10-14T01:19:33.649718+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:19:33.649718+0000","last_undegraded":"2023-10-14T01:19:33.649718+0000","last_fullsized":"2023-10-14T01:19:33.649718+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'7062","reported_seq":10690,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.649791+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:19:33.649791+0000","last_peered":"2023-10-14T01:19:33.649791+0000","last_clean":"2023-10-14T01:19:33.649791+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:19:33.649791+0000","last_undegraded":"2023-10-14T01:19:33.649791+0000","last_fullsized":"2023-10-14T01:19:33.649791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":179,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-14T01:18:10.751629+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:18:10.751629+0000","last_peered":"2023-10-14T01:18:10.751629+0000","last_clean":"2023-10-14T01:18:10.751629+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:18:10.751629+0000","last_undegraded":"2023-10-14T01:18:10.751629+0000","last_fullsized":"2023-10-14T01:18:10.751629+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"93'13947","reported_seq":19658,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.650210+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:19:33.650210+0000","last_peered":"2023-10-14T01:19:33.650210+0000","last_clean":"2023-10-14T01:19:33.650210+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:19:33.650210+0000","last_undegraded":"2023-10-14T01:19:33.650210+0000","last_fullsized":"2023-10-14T01:19:33.650210+0000","mapping_epoch":12,"log_start":"77'3900","ondisk_log_start":"77'3900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"93'505604","reported_seq":1003850,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:35.500716+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:19:35.500716+0000","last_peered":"2023-10-14T01:19:35.500716+0000","last_clean":"2023-10-14T01:19:35.500716+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:19:35.500716+0000","last_undegraded":"2023-10-14T01:19:35.500716+0000","last_fullsized":"2023-10-14T01:19:35.500716+0000","mapping_epoch":15,"log_start":"92'495534","ondisk_log_start":"92'495534","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7199776768,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":500259,"num_read_kb":1970576,"num_write":500431,"num_write_kb":2001724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"93'10397","reported_seq":14293,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.650048+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:19:33.650048+0000","last_peered":"2023-10-14T01:19:33.650048+0000","last_clean":"2023-10-14T01:19:33.650048+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:19:33.650048+0000","last_undegraded":"2023-10-14T01:19:33.650048+0000","last_fullsized":"2023-10-14T01:19:33.650048+0000","mapping_epoch":12,"log_start":"43'300","ondisk_log_start":"43'300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"93'513329","reported_seq":1019206,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:36.255812+0000","last_change":"2023-10-14T01:10:27.835457+0000","last_active":"2023-10-14T01:19:36.255812+0000","last_peered":"2023-10-14T01:19:36.255812+0000","last_clean":"2023-10-14T01:19:36.255812+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:19:36.255812+0000","last_undegraded":"2023-10-14T01:19:36.255812+0000","last_fullsized":"2023-10-14T01:19:36.255812+0000","mapping_epoch":15,"log_start":"92'503328","ondisk_log_start":"92'503328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3685,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 548s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7331831808,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":507936,"num_read_kb":2000568,"num_write":508179,"num_write_kb":2032716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4010,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:38.769900+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:19:38.769900+0000","last_peered":"2023-10-14T01:19:38.769900+0000","last_clean":"2023-10-14T01:19:38.769900+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:19:38.769900+0000","last_undegraded":"2023-10-14T01:19:38.769900+0000","last_fullsized":"2023-10-14T01:19:38.769900+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"93'516814","reported_seq":1026770,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:35.490120+0000","last_change":"2023-10-14T01:10:19.165908+0000","last_active":"2023-10-14T01:19:35.490120+0000","last_peered":"2023-10-14T01:19:35.490120+0000","last_clean":"2023-10-14T01:19:35.490120+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:19:35.490120+0000","last_undegraded":"2023-10-14T01:19:35.490120+0000","last_fullsized":"2023-10-14T01:19:35.490120+0000","mapping_epoch":15,"log_start":"92'506727","ondisk_log_start":"92'506727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3850,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 556s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7357132800,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":512024,"num_read_kb":2017464,"num_write":511542,"num_write_kb":2046168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1455,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.650531+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:19:33.650531+0000","last_peered":"2023-10-14T01:19:33.650531+0000","last_clean":"2023-10-14T01:19:33.650531+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:19:33.650531+0000","last_undegraded":"2023-10-14T01:19:33.650531+0000","last_fullsized":"2023-10-14T01:19:33.650531+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"93'511003","reported_seq":1018705,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:19:36.251203+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:19:36.251203+0000","last_peered":"2023-10-14T01:19:36.251203+0000","last_clean":"2023-10-14T01:19:36.251203+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:19:36.251203+0000","last_undegraded":"2023-10-14T01:19:36.251203+0000","last_fullsized":"2023-10-14T01:19:36.251203+0000","mapping_epoch":15,"log_start":"92'500934","ondisk_log_start":"92'500934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":232,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 30s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7335776275,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":509930,"num_read_kb":2008612,"num_write":508271,"num_write_kb":2033078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'13862","reported_seq":18923,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:33.650382+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:19:33.650382+0000","last_peered":"2023-10-14T01:19:33.650382+0000","last_clean":"2023-10-14T01:19:33.650382+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:19:33.650382+0000","last_undegraded":"2023-10-14T01:19:33.650382+0000","last_fullsized":"2023-10-14T01:19:33.650382+0000","mapping_epoch":12,"log_start":"77'3800","ondisk_log_start":"77'3800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:42.215993+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:18:42.215993+0000","last_peered":"2023-10-14T01:18:42.215993+0000","last_clean":"2023-10-14T01:18:42.215993+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:18:42.215993+0000","last_undegraded":"2023-10-14T01:18:42.215993+0000","last_fullsized":"2023-10-14T01:18:42.215993+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29224517651,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2030149,"num_read_kb":7997220,"num_write":2028423,"num_write_kb":8113686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16623542272,"data_stored":19691569190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40227,"ondisk_log_size":40227,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47601,"ondisk_log_size":47601,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739272,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10050032,"kb_used_data":8120936,"kb_used_omap":1268,"kb_used_meta":1927819,"kb_avail":84321808,"statfs":{"total":96636764160,"available":86345531392,"internally_reserved":0,"allocated":8315838464,"data_stored":9849334230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":1974086986},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.16700000000000001},"max":{"1min":1.242,"5min":1.242,"15min":2.5139999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.36799999999999999,"15min":0.376},"min":{"1min":0.20200000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.9449999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.26000000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.45000000000000001},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":1.2909999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.49399999999999999},"min":{"1min":0.27000000000000002,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":1.569,"5min":3.915,"15min":4.2060000000000004},"last":0.374}]}]},{"osd":1,"up_from":8,"seq":34359739272,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5998332,"kb_used_data":4066676,"kb_used_omap":510,"kb_used_meta":1931137,"kb_avail":88373508,"statfs":{"total":96636764160,"available":90494472192,"internally_reserved":0,"allocated":4164276224,"data_stored":4931351893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":523180,"internal_metadata":1977484372},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,9,8,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.53400000000000003,"15min":0.48999999999999999},"min":{"1min":0.191,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":1.9410000000000001,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.253},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.19600000000000001,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":5.3360000000000003,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:19:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.44800000000000001,"15min":0.41299999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.193},"max":{"1min":0.441,"5min":5.5890000000000004,"15min":5.6040000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.495,"15min":0.45800000000000002},"min":{"1min":0.27400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.47399999999999998,"5min":5.617,"15min":5.8529999999999998},"last":0.379}]}]},{"osd":0,"up_from":8,"seq":34359739274,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5957396,"kb_used_data":4049224,"kb_used_omap":502,"kb_used_meta":1907657,"kb_avail":88414444,"statfs":{"total":96636764160,"available":90536390656,"internally_reserved":0,"allocated":4146405376,"data_stored":4911931784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":514602,"internal_metadata":1953441238},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,8,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:18:52 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.47899999999999998,"15min":0.48099999999999998},"min":{"1min":0.20100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.39600000000000002,"5min":5.8600000000000003,"15min":8.3239999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.65700000000000003,"15min":0.52600000000000002},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.17399999999999999},"max":{"1min":2.4169999999999998,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.247}]},{"osd":2,"last update":"Sat Oct 14 01:18:52 2023","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.68000000000000005,"15min":0.63700000000000001},"min":{"1min":0.20200000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":0.68700000000000006,"5min":16.763999999999999,"15min":16.763999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.53900000000000003,"15min":0.497},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.49199999999999999,"5min":16.748000000000001,"15min":16.748000000000001},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4145053696,"data_stored":4911239168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4163379200,"data_stored":4931207187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8315109376,"data_stored":9849122835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:19:42.226 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:42.226 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:42.227 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:19:42.227 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:19:51.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:19:51.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:19:51.103 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:19:51.101+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:19:51.117 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:19:51.109+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:19:51.117 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:19:51.109+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:19:51.118 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:19:51.109+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:19:51.118 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:19:51.109+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:20:02.228 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:02.733 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:20:02.733 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:20:02.761 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2353,"stamp":"2023-10-14T01:20:02.454675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29265370956,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2062729,"num_read_kb":8056494,"num_write":2039180,"num_write_kb":8155731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87781,"ondisk_log_size":87781,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21237188,"kb_used_data":16308012,"kb_used_omap":2285,"kb_used_meta":4926866,"kb_avail":261878332,"statfs":{"total":289910292480,"available":268163411968,"internally_reserved":0,"allocated":16699404288,"data_stored":19765488799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2340267,"internal_metadata":5045111381},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,15,5,12,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24395776,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6168,"num_read_kb":24532,"num_write":6023,"num_write_kb":24092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002094"},"pg_stats":[{"pgid":"2.7","version":"94'10536","reported_seq":14561,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:00.931021+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:20:00.931021+0000","last_peered":"2023-10-14T01:20:00.931021+0000","last_clean":"2023-10-14T01:20:00.931021+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:20:00.931021+0000","last_undegraded":"2023-10-14T01:20:00.931021+0000","last_fullsized":"2023-10-14T01:20:00.931021+0000","mapping_epoch":12,"log_start":"47'500","ondisk_log_start":"47'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7092","reported_seq":10731,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:00.921980+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:20:00.921980+0000","last_peered":"2023-10-14T01:20:00.921980+0000","last_clean":"2023-10-14T01:20:00.921980+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:20:00.921980+0000","last_undegraded":"2023-10-14T01:20:00.921980+0000","last_fullsized":"2023-10-14T01:20:00.921980+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7092,"log_dups_size":0,"ondisk_log_size":7092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":180,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:18:42.207299+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:18:42.207299+0000","last_peered":"2023-10-14T01:18:42.207299+0000","last_clean":"2023-10-14T01:18:42.207299+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:18:42.207299+0000","last_undegraded":"2023-10-14T01:18:42.207299+0000","last_fullsized":"2023-10-14T01:18:42.207299+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'14006","reported_seq":19738,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:00.970300+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:20:00.970300+0000","last_peered":"2023-10-14T01:20:00.970300+0000","last_clean":"2023-10-14T01:20:00.970300+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:20:00.970300+0000","last_undegraded":"2023-10-14T01:20:00.970300+0000","last_fullsized":"2023-10-14T01:20:00.970300+0000","mapping_epoch":12,"log_start":"79'4000","ondisk_log_start":"79'4000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'508009","reported_seq":1008765,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:00.667055+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:20:00.667055+0000","last_peered":"2023-10-14T01:20:00.667055+0000","last_clean":"2023-10-14T01:20:00.667055+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:20:00.667055+0000","last_undegraded":"2023-10-14T01:20:00.667055+0000","last_fullsized":"2023-10-14T01:20:00.667055+0000","mapping_epoch":15,"log_start":"92'497934","ondisk_log_start":"92'497934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7209517056,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":502772,"num_read_kb":1980584,"num_write":502836,"num_write_kb":2011344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'10431","reported_seq":14343,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:19:59.253628+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:19:59.253628+0000","last_peered":"2023-10-14T01:19:59.253628+0000","last_clean":"2023-10-14T01:19:59.253628+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:19:59.253628+0000","last_undegraded":"2023-10-14T01:19:59.253628+0000","last_fullsized":"2023-10-14T01:19:59.253628+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'515904","reported_seq":1024397,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:01.437573+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:20:01.437573+0000","last_peered":"2023-10-14T01:20:01.437573+0000","last_clean":"2023-10-14T01:20:01.437573+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:20:01.437573+0000","last_undegraded":"2023-10-14T01:20:01.437573+0000","last_fullsized":"2023-10-14T01:20:01.437573+0000","mapping_epoch":15,"log_start":"92'505828","ondisk_log_start":"92'505828","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7341748224,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":510536,"num_read_kb":2010928,"num_write":510635,"num_write_kb":2042540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4024,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:19:59.263432+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:19:59.263432+0000","last_peered":"2023-10-14T01:19:59.263432+0000","last_clean":"2023-10-14T01:19:59.263432+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:19:59.263432+0000","last_undegraded":"2023-10-14T01:19:59.263432+0000","last_fullsized":"2023-10-14T01:19:59.263432+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'519324","reported_seq":1031835,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:00.655231+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:20:00.655231+0000","last_peered":"2023-10-14T01:20:00.655231+0000","last_clean":"2023-10-14T01:20:00.655231+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:20:00.655231+0000","last_undegraded":"2023-10-14T01:20:00.655231+0000","last_fullsized":"2023-10-14T01:20:00.655231+0000","mapping_epoch":15,"log_start":"92'509227","ondisk_log_start":"92'509227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7367204864,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":514575,"num_read_kb":2027592,"num_write":514041,"num_write_kb":2056164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1460,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:19:54.133219+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:19:54.133219+0000","last_peered":"2023-10-14T01:19:54.133219+0000","last_clean":"2023-10-14T01:19:54.133219+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:19:54.133219+0000","last_undegraded":"2023-10-14T01:19:54.133219+0000","last_fullsized":"2023-10-14T01:19:54.133219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'513740","reported_seq":1023939,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:01.441105+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:20:01.441105+0000","last_peered":"2023-10-14T01:20:01.441105+0000","last_clean":"2023-10-14T01:20:01.441105+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:20:01.441105+0000","last_undegraded":"2023-10-14T01:20:01.441105+0000","last_fullsized":"2023-10-14T01:20:01.441105+0000","mapping_epoch":15,"log_start":"92'503734","ondisk_log_start":"92'503734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":448,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 55s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7346307091,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":512409,"num_read_kb":2018476,"num_write":510858,"num_write_kb":2043426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'13921","reported_seq":19003,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:00.978571+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:20:00.978571+0000","last_peered":"2023-10-14T01:20:00.978571+0000","last_clean":"2023-10-14T01:20:00.978571+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:20:00.978571+0000","last_undegraded":"2023-10-14T01:20:00.978571+0000","last_fullsized":"2023-10-14T01:20:00.978571+0000","mapping_epoch":12,"log_start":"79'3900","ondisk_log_start":"79'3900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":269,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:11.032880+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:19:11.032880+0000","last_peered":"2023-10-14T01:19:11.032880+0000","last_clean":"2023-10-14T01:19:11.032880+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:19:11.032880+0000","last_undegraded":"2023-10-14T01:19:11.032880+0000","last_fullsized":"2023-10-14T01:19:11.032880+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29264777235,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2040292,"num_read_kb":8037580,"num_write":2038370,"num_write_kb":8153474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16696848384,"data_stored":19764875302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40254,"ondisk_log_size":40254,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47428,"ondisk_log_size":47428,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739276,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10167580,"kb_used_data":8152404,"kb_used_omap":1268,"kb_used_meta":2013899,"kb_avail":84204260,"statfs":{"total":96636764160,"available":86225162240,"internally_reserved":0,"allocated":8348061696,"data_stored":9881553018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2062232906},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.16700000000000001},"max":{"1min":1.242,"5min":1.242,"15min":2.5139999999999998},"last":0.22},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.36799999999999999,"15min":0.376},"min":{"1min":0.20200000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.9449999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.45000000000000001},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":1.2909999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.49399999999999999},"min":{"1min":0.27000000000000002,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":1.569,"5min":3.915,"15min":4.2060000000000004},"last":0.40500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739277,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5539684,"kb_used_data":4086876,"kb_used_omap":512,"kb_used_meta":1452287,"kb_avail":88832156,"statfs":{"total":96636764160,"available":90964127744,"internally_reserved":0,"allocated":4184961024,"data_stored":4952032249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1487142341},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,11,1,8,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:14 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.53400000000000003,"15min":0.48999999999999999},"min":{"1min":0.191,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":1.9410000000000001,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.19600000000000001,"5min":0.187,"15min":0.17100000000000001},"max":{"1min":5.3360000000000003,"5min":5.8879999999999999,"15min":8.8089999999999993},"last":0.20599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:19:14 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.44800000000000001,"15min":0.41299999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.193},"max":{"1min":0.441,"5min":5.5890000000000004,"15min":5.6040000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.495,"15min":0.45800000000000002},"min":{"1min":0.27400000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.47399999999999998,"5min":5.617,"15min":5.8529999999999998},"last":0.371}]}]},{"osd":0,"up_from":8,"seq":34359739279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5529924,"kb_used_data":4068732,"kb_used_omap":504,"kb_used_meta":1460679,"kb_avail":88841916,"statfs":{"total":96636764160,"available":90974121984,"internally_reserved":0,"allocated":4166381568,"data_stored":4931903532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1495736134},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.8220000000000001,"5min":4.8680000000000003,"15min":6.1130000000000004},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65200000000000002,"15min":0.52700000000000002},"min":{"1min":0.19700000000000001,"5min":0.193,"15min":0.17399999999999999},"max":{"1min":4.6260000000000003,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.215}]},{"osd":2,"last update":"Sat Oct 14 01:19:52 2023","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":1.008,"15min":0.73199999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":31.103000000000002,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.318},{"interface":"front","average":{"1min":1.978,"5min":0.85899999999999999,"15min":0.57399999999999995},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":30.234000000000002,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.23300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4165144576,"data_stored":4931330048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4184031232,"data_stored":4951859219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8347672576,"data_stored":9881686035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:20:02.763 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:02.764 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:06.474 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:20:06.475 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:20:06.475 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:20:06.476 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:20:06.476 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:20:06.476 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:20:06.476 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:20:06.477 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:20:06.477 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:20:06.477 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:20:06.477 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:20:06.478 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:20:06.478 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:06.473+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:20:11.543 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:20:11.544 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:20:11.544 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:20:11.544 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:20:11.545 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:20:11.545 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:20:11.545 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:20:11.545 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:20:11.546 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:20:11.546 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:20:11.546 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:20:11.546 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:20:11.547 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:11.537+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:20:22.765 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:23.275 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:20:23.276 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:20:23.328 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2363,"stamp":"2023-10-14T01:20:22.458106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29298593612,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2070976,"num_read_kb":8089310,"num_write":2047409,"num_write_kb":8188647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87816,"ondisk_log_size":87816,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21509248,"kb_used_data":16374312,"kb_used_omap":2285,"kb_used_meta":5132626,"kb_avail":261606272,"statfs":{"total":289910292480,"available":267884822528,"internally_reserved":0,"allocated":16767295488,"data_stored":19833379999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2340267,"internal_metadata":5255809621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,5,9,8,4,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25210880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6264,"num_read_kb":24924,"num_write":6245,"num_write_kb":24980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002058"},"pg_stats":[{"pgid":"2.7","version":"94'10602","reported_seq":14639,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:21.449152+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:20:21.449152+0000","last_peered":"2023-10-14T01:20:21.449152+0000","last_clean":"2023-10-14T01:20:21.449152+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:20:21.449152+0000","last_undegraded":"2023-10-14T01:20:21.449152+0000","last_fullsized":"2023-10-14T01:20:21.449152+0000","mapping_epoch":12,"log_start":"47'600","ondisk_log_start":"47'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7136","reported_seq":10783,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:21.438638+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:20:21.438638+0000","last_peered":"2023-10-14T01:20:21.438638+0000","last_clean":"2023-10-14T01:20:21.438638+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:20:21.438638+0000","last_undegraded":"2023-10-14T01:20:21.438638+0000","last_fullsized":"2023-10-14T01:20:21.438638+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7136,"log_dups_size":0,"ondisk_log_size":7136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":181,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:19:47.224502+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:19:47.224502+0000","last_peered":"2023-10-14T01:19:47.224502+0000","last_clean":"2023-10-14T01:19:47.224502+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:19:47.224502+0000","last_undegraded":"2023-10-14T01:19:47.224502+0000","last_fullsized":"2023-10-14T01:19:47.224502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'14094","reported_seq":19843,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:21.452078+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:20:21.452078+0000","last_peered":"2023-10-14T01:20:21.452078+0000","last_clean":"2023-10-14T01:20:21.452078+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:20:21.452078+0000","last_undegraded":"2023-10-14T01:20:21.452078+0000","last_fullsized":"2023-10-14T01:20:21.452078+0000","mapping_epoch":12,"log_start":"79'4000","ondisk_log_start":"79'4000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'510065","reported_seq":1012865,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:20.766665+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:20:20.766665+0000","last_peered":"2023-10-14T01:20:20.766665+0000","last_clean":"2023-10-14T01:20:20.766665+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:20:20.766665+0000","last_undegraded":"2023-10-14T01:20:20.766665+0000","last_fullsized":"2023-10-14T01:20:20.766665+0000","mapping_epoch":15,"log_start":"92'500034","ondisk_log_start":"92'500034","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7217856512,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":504809,"num_read_kb":1988696,"num_write":504892,"num_write_kb":2019568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'10479","reported_seq":14404,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:19.791513+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:20:19.791513+0000","last_peered":"2023-10-14T01:20:19.791513+0000","last_clean":"2023-10-14T01:20:19.791513+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:20:19.791513+0000","last_undegraded":"2023-10-14T01:20:19.791513+0000","last_fullsized":"2023-10-14T01:20:19.791513+0000","mapping_epoch":12,"log_start":"43'400","ondisk_log_start":"43'400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'517954","reported_seq":1028559,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:21.584602+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:20:21.584602+0000","last_peered":"2023-10-14T01:20:21.584602+0000","last_clean":"2023-10-14T01:20:21.584602+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:20:21.584602+0000","last_undegraded":"2023-10-14T01:20:21.584602+0000","last_fullsized":"2023-10-14T01:20:21.584602+0000","mapping_epoch":15,"log_start":"92'507928","ondisk_log_start":"92'507928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7350005760,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":512646,"num_read_kb":2019312,"num_write":512685,"num_write_kb":2050740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4036,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:19.731326+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:20:19.731326+0000","last_peered":"2023-10-14T01:20:19.731326+0000","last_clean":"2023-10-14T01:20:19.731326+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:20:19.731326+0000","last_undegraded":"2023-10-14T01:20:19.731326+0000","last_fullsized":"2023-10-14T01:20:19.731326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'521415","reported_seq":1035928,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:20.744926+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:20:20.744926+0000","last_peered":"2023-10-14T01:20:20.744926+0000","last_clean":"2023-10-14T01:20:20.744926+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:20:20.744926+0000","last_undegraded":"2023-10-14T01:20:20.744926+0000","last_fullsized":"2023-10-14T01:20:20.744926+0000","mapping_epoch":15,"log_start":"92'511327","ondisk_log_start":"92'511327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7375671296,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":516574,"num_read_kb":2035556,"num_write":516132,"num_write_kb":2064528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1464,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:14.610279+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:20:14.610279+0000","last_peered":"2023-10-14T01:20:14.610279+0000","last_clean":"2023-10-14T01:20:14.610279+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:20:14.610279+0000","last_undegraded":"2023-10-14T01:20:14.610279+0000","last_fullsized":"2023-10-14T01:20:14.610279+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'515844","reported_seq":1028156,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:21.592549+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:20:21.592549+0000","last_peered":"2023-10-14T01:20:21.592549+0000","last_clean":"2023-10-14T01:20:21.592549+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:20:21.592549+0000","last_undegraded":"2023-10-14T01:20:21.592549+0000","last_fullsized":"2023-10-14T01:20:21.592549+0000","mapping_epoch":15,"log_start":"92'505834","ondisk_log_start":"92'505834","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":553,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7354466323,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":514510,"num_read_kb":2026832,"num_write":512890,"num_write_kb":2051554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'14009","reported_seq":19108,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:21.466051+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:20:21.466051+0000","last_peered":"2023-10-14T01:20:21.466051+0000","last_clean":"2023-10-14T01:20:21.466051+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:20:21.466051+0000","last_undegraded":"2023-10-14T01:20:21.466051+0000","last_fullsized":"2023-10-14T01:20:21.466051+0000","mapping_epoch":12,"log_start":"79'4000","ondisk_log_start":"79'4000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":269,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:11.032880+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:19:11.032880+0000","last_peered":"2023-10-14T01:19:11.032880+0000","last_clean":"2023-10-14T01:19:11.032880+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:19:11.032880+0000","last_undegraded":"2023-10-14T01:19:11.032880+0000","last_fullsized":"2023-10-14T01:19:11.032880+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29297999891,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2048539,"num_read_kb":8070396,"num_write":2046599,"num_write_kb":8186390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16764231680,"data_stored":19832258598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40155,"ondisk_log_size":40155,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47562,"ondisk_log_size":47562,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739280,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10328252,"kb_used_data":8185076,"kb_used_omap":1268,"kb_used_meta":2141899,"kb_avail":84043588,"statfs":{"total":96636764160,"available":86060634112,"internally_reserved":0,"allocated":8381517824,"data_stored":9915009146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2193304906},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.191,"5min":0.17199999999999999,"15min":0.16700000000000001},"max":{"1min":1.242,"5min":1.242,"15min":2.5139999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.36799999999999999,"15min":0.376},"min":{"1min":0.20200000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":2.9449999999999998,"5min":3.7519999999999998,"15min":5.1109999999999998},"last":0.20599999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:19:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.45000000000000001},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.189},"max":{"1min":1.2909999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.49399999999999999},"min":{"1min":0.27000000000000002,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":1.569,"5min":3.915,"15min":4.2060000000000004},"last":0.28499999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739281,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5594240,"kb_used_data":4103928,"kb_used_omap":512,"kb_used_meta":1489791,"kb_avail":88777600,"statfs":{"total":96636764160,"available":90908262400,"internally_reserved":0,"allocated":4202422272,"data_stored":4969493497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1525546437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,5,3,4,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.55300000000000005,"15min":0.48099999999999998},"min":{"1min":0.20000000000000001,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":3.504,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":4.048,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.54100000000000004,"15min":0.436},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.58799999999999997,"15min":0.48199999999999998},"min":{"1min":0.28999999999999998,"5min":0.249,"15min":0.249},"max":{"1min":6.1059999999999999,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.35399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739283,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5586756,"kb_used_data":4085308,"kb_used_omap":504,"kb_used_meta":1500935,"kb_avail":88785084,"statfs":{"total":96636764160,"available":90915926016,"internally_reserved":0,"allocated":4183355392,"data_stored":4948877356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1536958278},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.8220000000000001,"5min":4.8680000000000003,"15min":6.1130000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65200000000000002,"15min":0.52700000000000002},"min":{"1min":0.19700000000000001,"5min":0.193,"15min":0.17399999999999999},"max":{"1min":4.6260000000000003,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":1.954}]},{"osd":2,"last update":"Sat Oct 14 01:19:52 2023","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":1.008,"15min":0.73199999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":31.103000000000002,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.19700000000000001},{"interface":"front","average":{"1min":1.978,"5min":0.85899999999999999,"15min":0.57399999999999995},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":30.234000000000002,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.39100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4182118400,"data_stored":4948303872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4200751104,"data_stored":4968579091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8381362176,"data_stored":9915375635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:20:23.330 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:23.330 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:30.418 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:30.420+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:20:30.419 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:30.420+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:20:30.419 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:30.420+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:20:30.420 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:30.420+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:20:43.331 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:20:43.818 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:20:43.819 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:20:43.852 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2373,"stamp":"2023-10-14T01:20:42.461429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29332021068,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2079034,"num_read_kb":8121342,"num_write":2055688,"num_write_kb":8221763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87949,"ondisk_log_size":87949,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21776896,"kb_used_data":16439720,"kb_used_omap":2285,"kb_used_meta":5334866,"kb_avail":261338624,"statfs":{"total":289910292480,"available":267610750976,"internally_reserved":0,"allocated":16834273280,"data_stored":19900357791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2340267,"internal_metadata":5462903381},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,6,10,5,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25169920,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6099,"num_read_kb":24232,"num_write":6233,"num_write_kb":24932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001970"},"pg_stats":[{"pgid":"2.7","version":"94'10658","reported_seq":14707,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.692290+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:20:40.692290+0000","last_peered":"2023-10-14T01:20:40.692290+0000","last_clean":"2023-10-14T01:20:40.692290+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:20:40.692290+0000","last_undegraded":"2023-10-14T01:20:40.692290+0000","last_fullsized":"2023-10-14T01:20:40.692290+0000","mapping_epoch":12,"log_start":"47'600","ondisk_log_start":"47'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"94'7174","reported_seq":10829,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.676540+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:20:40.676540+0000","last_peered":"2023-10-14T01:20:40.676540+0000","last_clean":"2023-10-14T01:20:40.676540+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:20:40.676540+0000","last_undegraded":"2023-10-14T01:20:40.676540+0000","last_fullsized":"2023-10-14T01:20:40.676540+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":181,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:19:47.224502+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:19:47.224502+0000","last_peered":"2023-10-14T01:19:47.224502+0000","last_clean":"2023-10-14T01:19:47.224502+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:19:47.224502+0000","last_undegraded":"2023-10-14T01:19:47.224502+0000","last_fullsized":"2023-10-14T01:19:47.224502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"94'14167","reported_seq":19932,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.694172+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:20:40.694172+0000","last_peered":"2023-10-14T01:20:40.694172+0000","last_clean":"2023-10-14T01:20:40.694172+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:20:40.694172+0000","last_undegraded":"2023-10-14T01:20:40.694172+0000","last_fullsized":"2023-10-14T01:20:40.694172+0000","mapping_epoch":12,"log_start":"79'4100","ondisk_log_start":"79'4100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"94'512056","reported_seq":1016848,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.810227+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:20:40.810227+0000","last_peered":"2023-10-14T01:20:40.810227+0000","last_clean":"2023-10-14T01:20:40.810227+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:20:40.810227+0000","last_undegraded":"2023-10-14T01:20:40.810227+0000","last_fullsized":"2023-10-14T01:20:40.810227+0000","mapping_epoch":15,"log_start":"93'502034","ondisk_log_start":"93'502034","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7225856000,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":506801,"num_read_kb":1996616,"num_write":506883,"num_write_kb":2027532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"94'10524","reported_seq":14461,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.794876+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:20:40.794876+0000","last_peered":"2023-10-14T01:20:40.794876+0000","last_clean":"2023-10-14T01:20:40.794876+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:20:40.794876+0000","last_undegraded":"2023-10-14T01:20:40.794876+0000","last_fullsized":"2023-10-14T01:20:40.794876+0000","mapping_epoch":12,"log_start":"47'500","ondisk_log_start":"47'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"94'520110","reported_seq":1032747,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:41.673047+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:20:41.673047+0000","last_peered":"2023-10-14T01:20:41.673047+0000","last_clean":"2023-10-14T01:20:41.673047+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:20:41.673047+0000","last_undegraded":"2023-10-14T01:20:41.673047+0000","last_fullsized":"2023-10-14T01:20:41.673047+0000","mapping_epoch":15,"log_start":"93'510028","ondisk_log_start":"93'510028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7358709760,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":514680,"num_read_kb":2027408,"num_write":514841,"num_write_kb":2059364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4048,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.245311+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:20:40.245311+0000","last_peered":"2023-10-14T01:20:40.245311+0000","last_clean":"2023-10-14T01:20:40.245311+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:20:40.245311+0000","last_undegraded":"2023-10-14T01:20:40.245311+0000","last_fullsized":"2023-10-14T01:20:40.245311+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"94'523437","reported_seq":1039972,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.814300+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:20:40.814300+0000","last_peered":"2023-10-14T01:20:40.814300+0000","last_clean":"2023-10-14T01:20:40.814300+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:20:40.814300+0000","last_undegraded":"2023-10-14T01:20:40.814300+0000","last_fullsized":"2023-10-14T01:20:40.814300+0000","mapping_epoch":15,"log_start":"93'513427","ondisk_log_start":"93'513427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7383846912,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":518598,"num_read_kb":2043608,"num_write":518154,"num_write_kb":2072616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1468,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:35.100134+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:20:35.100134+0000","last_peered":"2023-10-14T01:20:35.100134+0000","last_clean":"2023-10-14T01:20:35.100134+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:20:35.100134+0000","last_undegraded":"2023-10-14T01:20:35.100134+0000","last_fullsized":"2023-10-14T01:20:35.100134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"94'518023","reported_seq":1032353,"reported_epoch":94,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:20:41.693445+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:20:41.693445+0000","last_peered":"2023-10-14T01:20:41.693445+0000","last_clean":"2023-10-14T01:20:41.693445+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:20:41.693445+0000","last_undegraded":"2023-10-14T01:20:41.693445+0000","last_fullsized":"2023-10-14T01:20:41.693445+0000","mapping_epoch":15,"log_start":"93'507934","ondisk_log_start":"93'507934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":652,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 95s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7363014675,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":516518,"num_read_kb":2034796,"num_write":515000,"num_write_kb":2059994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"94'14082","reported_seq":19197,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:40.695643+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:20:40.695643+0000","last_peered":"2023-10-14T01:20:40.695643+0000","last_clean":"2023-10-14T01:20:40.695643+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:20:40.695643+0000","last_undegraded":"2023-10-14T01:20:40.695643+0000","last_fullsized":"2023-10-14T01:20:40.695643+0000","mapping_epoch":12,"log_start":"79'4000","ondisk_log_start":"79'4000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":269,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-14T01:19:11.032880+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:19:11.032880+0000","last_peered":"2023-10-14T01:19:11.032880+0000","last_clean":"2023-10-14T01:19:11.032880+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:19:11.032880+0000","last_undegraded":"2023-10-14T01:19:11.032880+0000","last_fullsized":"2023-10-14T01:19:11.032880+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29331427347,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2056597,"num_read_kb":8102428,"num_write":2054878,"num_write_kb":8219506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16831905792,"data_stored":19899932710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40203,"ondisk_log_size":40203,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47647,"ondisk_log_size":47647,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739284,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10465376,"kb_used_data":8218072,"kb_used_omap":1268,"kb_used_meta":2246027,"kb_avail":83906464,"statfs":{"total":96636764160,"available":85920219136,"internally_reserved":0,"allocated":8415305728,"data_stored":9948797050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299126,"internal_metadata":2299931978},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.34200000000000003,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.19,"15min":0.16700000000000001},"max":{"1min":0.372,"5min":1.242,"15min":2.5139999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.33400000000000002,"15min":0.36799999999999999},"min":{"1min":0.20599999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.443,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.218}]},{"osd":1,"last update":"Sat Oct 14 01:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.189},"max":{"1min":3.5529999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":3.5950000000000002,"5min":3.915,"15min":4.2060000000000004},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5664692,"kb_used_data":4120236,"kb_used_omap":512,"kb_used_meta":1543935,"kb_avail":88707148,"statfs":{"total":96636764160,"available":90836119552,"internally_reserved":0,"allocated":4219121664,"data_stored":4986192889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1580989893},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,8,3,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.55300000000000005,"15min":0.48099999999999998},"min":{"1min":0.20000000000000001,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":3.504,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.308},{"interface":"front","average":{"1min":0.498,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":4.048,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.21099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.54100000000000004,"15min":0.436},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.58799999999999997,"15min":0.48199999999999998},"min":{"1min":0.28999999999999998,"5min":0.249,"15min":0.249},"max":{"1min":6.1059999999999999,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739287,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5646828,"kb_used_data":4101412,"kb_used_omap":504,"kb_used_meta":1544903,"kb_avail":88725012,"statfs":{"total":96636764160,"available":90854412288,"internally_reserved":0,"allocated":4199845888,"data_stored":4965367852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1581981510},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:19:52 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.51700000000000002,"15min":0.47099999999999997},"min":{"1min":0.21199999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.8220000000000001,"5min":4.8680000000000003,"15min":6.1130000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65200000000000002,"15min":0.52700000000000002},"min":{"1min":0.19700000000000001,"5min":0.193,"15min":0.17399999999999999},"max":{"1min":4.6260000000000003,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.216}]},{"osd":2,"last update":"Sat Oct 14 01:19:52 2023","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":1.008,"15min":0.73199999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":31.103000000000002,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.317},{"interface":"front","average":{"1min":1.978,"5min":0.85899999999999999,"15min":0.57399999999999995},"min":{"1min":0.20999999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":30.234000000000002,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4198551552,"data_stored":4964737024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4218224640,"data_stored":4986052627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8415129600,"data_stored":9949143059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:20:43.854 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:20:43.854 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:20:57.569 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:57.560+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:20:57.570 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:57.564+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:20:57.571 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:57.568+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:20:57.577 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:20:57.576+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:21:01.337 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:01.332+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:21:01.338 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:01.332+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:21:03.855 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:04.338 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:21:04.339 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:21:04.367 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2384,"stamp":"2023-10-14T01:21:02.464803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29365743436,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2087476,"num_read_kb":8154906,"num_write":2064043,"num_write_kb":8255183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87938,"ondisk_log_size":87938,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21041192,"kb_used_data":16506316,"kb_used_omap":2287,"kb_used_meta":4532560,"kb_avail":262074328,"statfs":{"total":289910292480,"available":268364111872,"internally_reserved":0,"allocated":16902467584,"data_stored":19968538634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":4641342405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,4,17,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16957440,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4226,"num_read_kb":16820,"num_write":4205,"num_write_kb":16820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001676"},"pg_stats":[{"pgid":"2.7","version":"95'10683","reported_seq":14746,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.364605+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:21:01.364605+0000","last_peered":"2023-10-14T01:21:01.364605+0000","last_clean":"2023-10-14T01:21:01.364605+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:21:01.364605+0000","last_undegraded":"2023-10-14T01:21:01.364605+0000","last_fullsized":"2023-10-14T01:21:01.364605+0000","mapping_epoch":12,"log_start":"47'600","ondisk_log_start":"47'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7191","reported_seq":10856,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.348393+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:21:01.348393+0000","last_peered":"2023-10-14T01:21:01.348393+0000","last_clean":"2023-10-14T01:21:01.348393+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:21:01.348393+0000","last_undegraded":"2023-10-14T01:21:01.348393+0000","last_fullsized":"2023-10-14T01:21:01.348393+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7191,"log_dups_size":0,"ondisk_log_size":7191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":182,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:16.592389+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:20:16.592389+0000","last_peered":"2023-10-14T01:20:16.592389+0000","last_clean":"2023-10-14T01:20:16.592389+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:20:16.592389+0000","last_undegraded":"2023-10-14T01:20:16.592389+0000","last_fullsized":"2023-10-14T01:20:16.592389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'14200","reported_seq":19982,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.357703+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:21:01.357703+0000","last_peered":"2023-10-14T01:21:01.357703+0000","last_clean":"2023-10-14T01:21:01.357703+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:21:01.357703+0000","last_undegraded":"2023-10-14T01:21:01.357703+0000","last_fullsized":"2023-10-14T01:21:01.357703+0000","mapping_epoch":12,"log_start":"79'4100","ondisk_log_start":"79'4100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'514144","reported_seq":1020954,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:00.891491+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:21:00.891491+0000","last_peered":"2023-10-14T01:21:00.891491+0000","last_clean":"2023-10-14T01:21:00.891491+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:21:00.891491+0000","last_undegraded":"2023-10-14T01:21:00.891491+0000","last_fullsized":"2023-10-14T01:21:00.891491+0000","mapping_epoch":15,"log_start":"93'504134","ondisk_log_start":"93'504134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7234240512,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":508818,"num_read_kb":2004616,"num_write":508971,"num_write_kb":2035884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'10563","reported_seq":14513,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:00.689899+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:21:00.689899+0000","last_peered":"2023-10-14T01:21:00.689899+0000","last_clean":"2023-10-14T01:21:00.689899+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:21:00.689899+0000","last_undegraded":"2023-10-14T01:21:00.689899+0000","last_fullsized":"2023-10-14T01:21:00.689899+0000","mapping_epoch":12,"log_start":"47'500","ondisk_log_start":"47'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'522179","reported_seq":1036927,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.779370+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:21:01.779370+0000","last_peered":"2023-10-14T01:21:01.779370+0000","last_clean":"2023-10-14T01:21:01.779370+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:21:01.779370+0000","last_undegraded":"2023-10-14T01:21:01.779370+0000","last_fullsized":"2023-10-14T01:21:01.779370+0000","mapping_epoch":15,"log_start":"93'512128","ondisk_log_start":"93'512128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7367057408,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":516790,"num_read_kb":2035812,"num_write":516910,"num_write_kb":2067640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4058,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:55.574371+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:20:55.574371+0000","last_peered":"2023-10-14T01:20:55.574371+0000","last_clean":"2023-10-14T01:20:55.574371+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:20:55.574371+0000","last_undegraded":"2023-10-14T01:20:55.574371+0000","last_fullsized":"2023-10-14T01:20:55.574371+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'525534","reported_seq":1044187,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:00.899584+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:21:00.899584+0000","last_peered":"2023-10-14T01:21:00.899584+0000","last_clean":"2023-10-14T01:21:00.899584+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:21:00.899584+0000","last_undegraded":"2023-10-14T01:21:00.899584+0000","last_fullsized":"2023-10-14T01:21:00.899584+0000","mapping_epoch":15,"log_start":"93'515527","ondisk_log_start":"93'515527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7392321536,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":520717,"num_read_kb":2052032,"num_write":520251,"num_write_kb":2081004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1472,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:55.573752+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:20:55.573752+0000","last_peered":"2023-10-14T01:20:55.573752+0000","last_clean":"2023-10-14T01:20:55.573752+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:20:55.573752+0000","last_undegraded":"2023-10-14T01:20:55.573752+0000","last_fullsized":"2023-10-14T01:20:55.573752+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'520211","reported_seq":1036752,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:01.797231+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:21:01.797231+0000","last_peered":"2023-10-14T01:21:01.797231+0000","last_clean":"2023-10-14T01:21:01.797231+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:21:01.797231+0000","last_undegraded":"2023-10-14T01:21:01.797231+0000","last_fullsized":"2023-10-14T01:21:01.797231+0000","mapping_epoch":15,"log_start":"93'510134","ondisk_log_start":"93'510134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":778,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 115s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7371530259,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":518714,"num_read_kb":2043532,"num_write":517101,"num_write_kb":2068398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'14115","reported_seq":19247,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:01.379914+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:21:01.379914+0000","last_peered":"2023-10-14T01:21:01.379914+0000","last_clean":"2023-10-14T01:21:01.379914+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:21:01.379914+0000","last_undegraded":"2023-10-14T01:21:01.379914+0000","last_fullsized":"2023-10-14T01:21:01.379914+0000","mapping_epoch":12,"log_start":"81'4100","ondisk_log_start":"81'4100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":270,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:19:47.317560+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:19:47.317560+0000","last_peered":"2023-10-14T01:19:47.317560+0000","last_clean":"2023-10-14T01:19:47.317560+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:19:47.317560+0000","last_undegraded":"2023-10-14T01:19:47.317560+0000","last_fullsized":"2023-10-14T01:19:47.317560+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29365149715,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2065039,"num_read_kb":8135992,"num_write":2063233,"num_write_kb":8252926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16900300800,"data_stored":19968327718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40145,"ondisk_log_size":40145,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47694,"ondisk_log_size":47694,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739288,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9613556,"kb_used_data":8251304,"kb_used_omap":1270,"kb_used_meta":1360969,"kb_avail":84758284,"statfs":{"total":96636764160,"available":86792482816,"internally_reserved":0,"allocated":8449335296,"data_stored":9982822131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1393632954},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.34200000000000003,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.19,"15min":0.16700000000000001},"max":{"1min":0.372,"5min":1.242,"15min":2.5139999999999998},"last":0.17999999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.33400000000000002,"15min":0.36799999999999999},"min":{"1min":0.20599999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.443,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.32000000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.189},"max":{"1min":3.5529999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":3.5950000000000002,"5min":3.915,"15min":4.2060000000000004},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5721736,"kb_used_data":4137024,"kb_used_omap":512,"kb_used_meta":1584191,"kb_avail":88650104,"statfs":{"total":96636764160,"available":90777706496,"internally_reserved":0,"allocated":4236312576,"data_stored":5003379314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1622212037},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,12,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.55300000000000005,"15min":0.48099999999999998},"min":{"1min":0.20000000000000001,"5min":0.16500000000000001,"15min":0.152},"max":{"1min":3.504,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":4.048,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.24399999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.54100000000000004,"15min":0.436},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":6.0709999999999997,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.58799999999999997,"15min":0.48199999999999998},"min":{"1min":0.28999999999999998,"5min":0.249,"15min":0.249},"max":{"1min":6.1059999999999999,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.36499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5705900,"kb_used_data":4117988,"kb_used_omap":504,"kb_used_meta":1587399,"kb_avail":88665940,"statfs":{"total":96636764160,"available":90793922560,"internally_reserved":0,"allocated":4216819712,"data_stored":4982337189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1625497414},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.60999999999999999,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":1.954,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.224}]},{"osd":2,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":1.012,"15min":0.73799999999999999},"min":{"1min":0.182,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":1.8460000000000001,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.85399999999999998,"15min":0.57299999999999995},"min":{"1min":0.23300000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47099999999999997,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4215693312,"data_stored":4981878784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4235317248,"data_stored":5003145235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8449290240,"data_stored":9983303699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:21:04.369 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:04.370 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:04.468 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:04.464+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:21:04.469 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:04.468+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:21:04.469 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:04.468+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:21:04.470 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:04.472+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:21:04.471 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:04.472+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:21:04.471 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:04.472+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:21:05.905 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:05.904+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:21:05.912 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:21:05.912+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:21:24.371 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:24.966 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:21:24.966 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:21:25.005 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2395,"stamp":"2023-10-14T01:21:24.468657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29399523148,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2095803,"num_read_kb":8188054,"num_write":2072411,"num_write_kb":8288655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88102,"ondisk_log_size":88102,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21293448,"kb_used_data":16572464,"kb_used_omap":2287,"kb_used_meta":4718672,"kb_avail":261822072,"statfs":{"total":289910292480,"available":268105801728,"internally_reserved":0,"allocated":16970203136,"data_stored":20036274186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":4831921093},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,3,17,12,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17145856,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4217,"num_read_kb":16784,"num_write":4254,"num_write_kb":17016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"95'10725","reported_seq":14800,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:21.173623+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:21:21.173623+0000","last_peered":"2023-10-14T01:21:21.173623+0000","last_clean":"2023-10-14T01:21:21.173623+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:21:21.173623+0000","last_undegraded":"2023-10-14T01:21:21.173623+0000","last_fullsized":"2023-10-14T01:21:21.173623+0000","mapping_epoch":12,"log_start":"49'700","ondisk_log_start":"49'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7220","reported_seq":10893,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:21.173233+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:21:21.173233+0000","last_peered":"2023-10-14T01:21:21.173233+0000","last_clean":"2023-10-14T01:21:21.173233+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:21:21.173233+0000","last_undegraded":"2023-10-14T01:21:21.173233+0000","last_fullsized":"2023-10-14T01:21:21.173233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7220,"log_dups_size":0,"ondisk_log_size":7220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":182,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:16.592389+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:20:16.592389+0000","last_peered":"2023-10-14T01:20:16.592389+0000","last_clean":"2023-10-14T01:20:16.592389+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:20:16.592389+0000","last_undegraded":"2023-10-14T01:20:16.592389+0000","last_fullsized":"2023-10-14T01:20:16.592389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'14255","reported_seq":20053,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:21.173650+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:21:21.173650+0000","last_peered":"2023-10-14T01:21:21.173650+0000","last_clean":"2023-10-14T01:21:21.173650+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:21:21.173650+0000","last_undegraded":"2023-10-14T01:21:21.173650+0000","last_fullsized":"2023-10-14T01:21:21.173650+0000","mapping_epoch":12,"log_start":"81'4200","ondisk_log_start":"81'4200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'516224","reported_seq":1025057,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:20.942209+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:21:20.942209+0000","last_peered":"2023-10-14T01:21:20.942209+0000","last_clean":"2023-10-14T01:21:20.942209+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:21:20.942209+0000","last_undegraded":"2023-10-14T01:21:20.942209+0000","last_fullsized":"2023-10-14T01:21:20.942209+0000","mapping_epoch":15,"log_start":"93'506134","ondisk_log_start":"93'506134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7242649600,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":510840,"num_read_kb":2012656,"num_write":511051,"num_write_kb":2044204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'10599","reported_seq":14558,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:20.411819+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:21:20.411819+0000","last_peered":"2023-10-14T01:21:20.411819+0000","last_clean":"2023-10-14T01:21:20.411819+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:21:20.411819+0000","last_undegraded":"2023-10-14T01:21:20.411819+0000","last_fullsized":"2023-10-14T01:21:20.411819+0000","mapping_epoch":12,"log_start":"47'500","ondisk_log_start":"47'500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'524261","reported_seq":1041136,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:21.922453+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:21:21.922453+0000","last_peered":"2023-10-14T01:21:21.922453+0000","last_clean":"2023-10-14T01:21:21.922453+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:21:21.922453+0000","last_undegraded":"2023-10-14T01:21:21.922453+0000","last_fullsized":"2023-10-14T01:21:21.922453+0000","mapping_epoch":15,"log_start":"93'514228","ondisk_log_start":"93'514228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7375437824,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":518917,"num_read_kb":2044272,"num_write":518992,"num_write_kb":2075968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4070,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:16.051233+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:21:16.051233+0000","last_peered":"2023-10-14T01:21:16.051233+0000","last_clean":"2023-10-14T01:21:16.051233+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:21:16.051233+0000","last_undegraded":"2023-10-14T01:21:16.051233+0000","last_fullsized":"2023-10-14T01:21:16.051233+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'527618","reported_seq":1048361,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:20.937373+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:21:20.937373+0000","last_peered":"2023-10-14T01:21:20.937373+0000","last_clean":"2023-10-14T01:21:20.937373+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:21:20.937373+0000","last_undegraded":"2023-10-14T01:21:20.937373+0000","last_fullsized":"2023-10-14T01:21:20.937373+0000","mapping_epoch":15,"log_start":"93'517527","ondisk_log_start":"93'517527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7400685568,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":522806,"num_read_kb":2060364,"num_write":522335,"num_write_kb":2089340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1477,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:15.373447+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:21:15.373447+0000","last_peered":"2023-10-14T01:21:15.373447+0000","last_clean":"2023-10-14T01:21:15.373447+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:21:15.373447+0000","last_undegraded":"2023-10-14T01:21:15.373447+0000","last_fullsized":"2023-10-14T01:21:15.373447+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'522412","reported_seq":1041053,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:21.924863+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:21:21.924863+0000","last_peered":"2023-10-14T01:21:21.924863+0000","last_clean":"2023-10-14T01:21:21.924863+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:21:21.924863+0000","last_undegraded":"2023-10-14T01:21:21.924863+0000","last_fullsized":"2023-10-14T01:21:21.924863+0000","mapping_epoch":15,"log_start":"94'512334","ondisk_log_start":"94'512334","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":889,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 135s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7380156435,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":520803,"num_read_kb":2051848,"num_write":519223,"num_write_kb":2076886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'14170","reported_seq":19318,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:21.174077+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:21:21.174077+0000","last_peered":"2023-10-14T01:21:21.174077+0000","last_clean":"2023-10-14T01:21:21.174077+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:21:21.174077+0000","last_undegraded":"2023-10-14T01:21:21.174077+0000","last_fullsized":"2023-10-14T01:21:21.174077+0000","mapping_epoch":12,"log_start":"81'4100","ondisk_log_start":"81'4100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:20:57.270008+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:20:57.270008+0000","last_peered":"2023-10-14T01:20:57.270008+0000","last_clean":"2023-10-14T01:20:57.270008+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:20:57.270008+0000","last_undegraded":"2023-10-14T01:20:57.270008+0000","last_fullsized":"2023-10-14T01:20:57.270008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29398929427,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2073366,"num_read_kb":8169140,"num_write":2071601,"num_write_kb":8286398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16968720384,"data_stored":20036747302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40292,"ondisk_log_size":40292,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47711,"ondisk_log_size":47711,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739292,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9742440,"kb_used_data":8284320,"kb_used_omap":1270,"kb_used_meta":1456841,"kb_avail":84629400,"statfs":{"total":96636764160,"available":86660505600,"internally_reserved":0,"allocated":8483143680,"data_stored":10016630515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1491805882},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.34200000000000003,"15min":0.33500000000000002},"min":{"1min":0.19900000000000001,"5min":0.19,"15min":0.16700000000000001},"max":{"1min":0.372,"5min":1.242,"15min":2.5139999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.33400000000000002,"15min":0.36799999999999999},"min":{"1min":0.20599999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.443,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.22600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:20:36 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.22500000000000001,"5min":0.223,"15min":0.189},"max":{"1min":3.5529999999999999,"5min":3.883,"15min":4.1840000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55500000000000005,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.27000000000000002,"15min":0.23799999999999999},"max":{"1min":3.5950000000000002,"5min":3.915,"15min":4.2060000000000004},"last":0.501}]}]},{"osd":1,"up_from":8,"seq":34359739293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5774548,"kb_used_data":4153740,"kb_used_omap":512,"kb_used_meta":1620287,"kb_avail":88597292,"statfs":{"total":96636764160,"available":90723627008,"internally_reserved":0,"allocated":4253429760,"data_stored":5020496498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1659174341},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,9,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.53800000000000003,"15min":0.48099999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.152},"max":{"1min":0.70199999999999996,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":3.891},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.21099999999999999,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.72699999999999998,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:21:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.52200000000000002,"15min":0.42699999999999999},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":1.3999999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57099999999999995,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.249},"max":{"1min":1.4690000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.37}]}]},{"osd":0,"up_from":8,"seq":34359739295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5776460,"kb_used_data":4134404,"kb_used_omap":504,"kb_used_meta":1641543,"kb_avail":88595380,"statfs":{"total":96636764160,"available":90721669120,"internally_reserved":0,"allocated":4233629696,"data_stored":4999147173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1680940870},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,8,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.60999999999999999,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":1.954,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.68700000000000006}]},{"osd":2,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":1.012,"15min":0.73799999999999999},"min":{"1min":0.182,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":1.8460000000000001,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.85399999999999998,"15min":0.57299999999999995},"min":{"1min":0.23300000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47099999999999997,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4232761344,"data_stored":4998946816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4252528640,"data_stored":5020356627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8483430400,"data_stored":10017443859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:21:25.007 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:25.008 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:21:45.009 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:21:45.890 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:21:45.891 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:21:45.927 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2405,"stamp":"2023-10-14T01:21:44.472026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29433007948,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2104197,"num_read_kb":8221462,"num_write":2080734,"num_write_kb":8321947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87959,"ondisk_log_size":87959,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21533672,"kb_used_data":16639696,"kb_used_omap":2287,"kb_used_meta":4891664,"kb_avail":261581848,"statfs":{"total":289910292480,"available":267859812352,"internally_reserved":0,"allocated":17039048704,"data_stored":20105119754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":5009064901},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,2,5,8,13,8,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16883712,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4220,"num_read_kb":16788,"num_write":4189,"num_write_kb":16756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002116"},"pg_stats":[{"pgid":"2.7","version":"95'10768","reported_seq":14855,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:41.649778+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:21:41.649778+0000","last_peered":"2023-10-14T01:21:41.649778+0000","last_clean":"2023-10-14T01:21:41.649778+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:21:41.649778+0000","last_undegraded":"2023-10-14T01:21:41.649778+0000","last_fullsized":"2023-10-14T01:21:41.649778+0000","mapping_epoch":12,"log_start":"49'700","ondisk_log_start":"49'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7250","reported_seq":10931,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:41.649593+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:21:41.649593+0000","last_peered":"2023-10-14T01:21:41.649593+0000","last_clean":"2023-10-14T01:21:41.649593+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:21:41.649593+0000","last_undegraded":"2023-10-14T01:21:41.649593+0000","last_fullsized":"2023-10-14T01:21:41.649593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7250,"log_dups_size":0,"ondisk_log_size":7250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":182,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:16.592389+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:20:16.592389+0000","last_peered":"2023-10-14T01:20:16.592389+0000","last_clean":"2023-10-14T01:20:16.592389+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:20:16.592389+0000","last_undegraded":"2023-10-14T01:20:16.592389+0000","last_fullsized":"2023-10-14T01:20:16.592389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'14313","reported_seq":20127,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:41.650410+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:21:41.650410+0000","last_peered":"2023-10-14T01:21:41.650410+0000","last_clean":"2023-10-14T01:21:41.650410+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:21:41.650410+0000","last_undegraded":"2023-10-14T01:21:41.650410+0000","last_fullsized":"2023-10-14T01:21:41.650410+0000","mapping_epoch":12,"log_start":"81'4300","ondisk_log_start":"81'4300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'518234","reported_seq":1029107,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:41.016004+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:21:41.016004+0000","last_peered":"2023-10-14T01:21:41.016004+0000","last_clean":"2023-10-14T01:21:41.016004+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:21:41.016004+0000","last_undegraded":"2023-10-14T01:21:41.016004+0000","last_fullsized":"2023-10-14T01:21:41.016004+0000","mapping_epoch":15,"log_start":"94'508134","ondisk_log_start":"94'508134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7250763776,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":512881,"num_read_kb":2020768,"num_write":513061,"num_write_kb":2052244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'10651","reported_seq":14622,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:40.045830+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:21:40.045830+0000","last_peered":"2023-10-14T01:21:40.045830+0000","last_clean":"2023-10-14T01:21:40.045830+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:21:40.045830+0000","last_undegraded":"2023-10-14T01:21:40.045830+0000","last_fullsized":"2023-10-14T01:21:40.045830+0000","mapping_epoch":12,"log_start":"47'600","ondisk_log_start":"47'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'526337","reported_seq":1045389,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:42.086534+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:21:42.086534+0000","last_peered":"2023-10-14T01:21:42.086534+0000","last_clean":"2023-10-14T01:21:42.086534+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:21:42.086534+0000","last_undegraded":"2023-10-14T01:21:42.086534+0000","last_fullsized":"2023-10-14T01:21:42.086534+0000","mapping_epoch":15,"log_start":"94'516328","ondisk_log_start":"94'516328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7383760896,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":521093,"num_read_kb":2052924,"num_write":521068,"num_write_kb":2084272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4082,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:36.530078+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:21:36.530078+0000","last_peered":"2023-10-14T01:21:36.530078+0000","last_clean":"2023-10-14T01:21:36.530078+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:21:36.530078+0000","last_undegraded":"2023-10-14T01:21:36.530078+0000","last_fullsized":"2023-10-14T01:21:36.530078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'529749","reported_seq":1052586,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:40.994636+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:21:40.994636+0000","last_peered":"2023-10-14T01:21:40.994636+0000","last_clean":"2023-10-14T01:21:40.994636+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:21:40.994636+0000","last_undegraded":"2023-10-14T01:21:40.994636+0000","last_fullsized":"2023-10-14T01:21:40.994636+0000","mapping_epoch":15,"log_start":"94'519727","ondisk_log_start":"94'519727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7409238016,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":524898,"num_read_kb":2068692,"num_write":524466,"num_write_kb":2097864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1481,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:31.417369+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:21:31.417369+0000","last_peered":"2023-10-14T01:21:31.417369+0000","last_clean":"2023-10-14T01:21:31.417369+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:21:31.417369+0000","last_undegraded":"2023-10-14T01:21:31.417369+0000","last_fullsized":"2023-10-14T01:21:31.417369+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'524612","reported_seq":1045350,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:21:42.086704+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:21:42.086704+0000","last_peered":"2023-10-14T01:21:42.086704+0000","last_clean":"2023-10-14T01:21:42.086704+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:21:42.086704+0000","last_undegraded":"2023-10-14T01:21:42.086704+0000","last_fullsized":"2023-10-14T01:21:42.086704+0000","mapping_epoch":15,"log_start":"94'514534","ondisk_log_start":"94'514534","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1030,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 156s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7388651539,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":522888,"num_read_kb":2060164,"num_write":521329,"num_write_kb":2085310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'14227","reported_seq":19391,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:41.651032+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:21:41.651032+0000","last_peered":"2023-10-14T01:21:41.651032+0000","last_clean":"2023-10-14T01:21:41.651032+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:21:41.651032+0000","last_undegraded":"2023-10-14T01:21:41.651032+0000","last_fullsized":"2023-10-14T01:21:41.651032+0000","mapping_epoch":12,"log_start":"81'4200","ondisk_log_start":"81'4200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:20:57.270008+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:20:57.270008+0000","last_peered":"2023-10-14T01:20:57.270008+0000","last_clean":"2023-10-14T01:20:57.270008+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:20:57.270008+0000","last_undegraded":"2023-10-14T01:20:57.270008+0000","last_fullsized":"2023-10-14T01:20:57.270008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29432414227,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2081760,"num_read_kb":8202548,"num_write":2079924,"num_write_kb":8319690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17036795904,"data_stored":20104822822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40209,"ondisk_log_size":40209,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47651,"ondisk_log_size":47651,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739296,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9847756,"kb_used_data":8317444,"kb_used_omap":1270,"kb_used_meta":1529033,"kb_avail":84524084,"statfs":{"total":96636764160,"available":86552662016,"internally_reserved":0,"allocated":8517062656,"data_stored":10050549491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1565730490},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.40300000000000002,"15min":0.35399999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.33100000000000002,"15min":0.36199999999999999},"min":{"1min":0.218,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.40699999999999997,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.28000000000000003}]},{"osd":1,"last update":"Sat Oct 14 01:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.57599999999999996,"15min":0.48499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.63200000000000001,"15min":0.53300000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.8449999999999998,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.30399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739297,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856728,"kb_used_data":4171216,"kb_used_omap":512,"kb_used_meta":1684991,"kb_avail":88515112,"statfs":{"total":96636764160,"available":90639474688,"internally_reserved":0,"allocated":4271325184,"data_stored":5038391922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1725431237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,1,1,10,7,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.53800000000000003,"15min":0.48099999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.152},"max":{"1min":0.70199999999999996,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.21099999999999999,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.72699999999999998,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.51800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:21:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.52200000000000002,"15min":0.42699999999999999},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":1.3999999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57099999999999995,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.249},"max":{"1min":1.4690000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.29799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5829188,"kb_used_data":4151036,"kb_used_omap":504,"kb_used_meta":1677639,"kb_avail":88542652,"statfs":{"total":96636764160,"available":90667675648,"internally_reserved":0,"allocated":4250660864,"data_stored":5016178341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1717903174},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,0,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.60999999999999999,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":1.954,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.20000000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":1.012,"15min":0.73799999999999999},"min":{"1min":0.182,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":1.8460000000000001,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.85399999999999998,"15min":0.57299999999999995},"min":{"1min":0.23300000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47099999999999997,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4249710592,"data_stored":5015896064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4269654016,"data_stored":5037482003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8517431296,"data_stored":10051444755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:21:45.929 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:21:45.930 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:02.019 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.018+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:22:02.032 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.030+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:22:02.033 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.030+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:22:02.033 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.030+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:22:02.033 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.030+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:22:02.034 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.030+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:22:02.034 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:02.030+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:22:05.930 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:06.380 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:22:06.382 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:22:06.411 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2415,"stamp":"2023-10-14T01:22:04.475359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29465866060,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2112259,"num_read_kb":8253554,"num_write":2088862,"num_write_kb":8354459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87978,"ondisk_log_size":87978,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21686228,"kb_used_data":16704828,"kb_used_omap":2287,"kb_used_meta":4979088,"kb_avail":261429292,"statfs":{"total":289910292480,"available":267703595008,"internally_reserved":0,"allocated":17105743872,"data_stored":20171814922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":5098587077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,5,5,6,9,24],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15994880,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3932,"num_read_kb":15660,"num_write":3958,"num_write_kb":15832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001979"},"pg_stats":[{"pgid":"2.7","version":"95'10807","reported_seq":14906,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:02.129730+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:22:02.129730+0000","last_peered":"2023-10-14T01:22:02.129730+0000","last_clean":"2023-10-14T01:22:02.129730+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:22:02.129730+0000","last_undegraded":"2023-10-14T01:22:02.129730+0000","last_fullsized":"2023-10-14T01:22:02.129730+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"95'7276","reported_seq":10965,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:02.129516+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:22:02.129516+0000","last_peered":"2023-10-14T01:22:02.129516+0000","last_clean":"2023-10-14T01:22:02.129516+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:22:02.129516+0000","last_undegraded":"2023-10-14T01:22:02.129516+0000","last_fullsized":"2023-10-14T01:22:02.129516+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":182,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-10-14T01:20:16.592389+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:20:16.592389+0000","last_peered":"2023-10-14T01:20:16.592389+0000","last_clean":"2023-10-14T01:20:16.592389+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:20:16.592389+0000","last_undegraded":"2023-10-14T01:20:16.592389+0000","last_fullsized":"2023-10-14T01:20:16.592389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"95'14363","reported_seq":20193,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:02.129867+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:22:02.129867+0000","last_peered":"2023-10-14T01:22:02.129867+0000","last_clean":"2023-10-14T01:22:02.129867+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:22:02.129867+0000","last_undegraded":"2023-10-14T01:22:02.129867+0000","last_fullsized":"2023-10-14T01:22:02.129867+0000","mapping_epoch":12,"log_start":"81'4300","ondisk_log_start":"81'4300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"95'520175","reported_seq":1032976,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:01.025611+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:22:01.025611+0000","last_peered":"2023-10-14T01:22:01.025611+0000","last_clean":"2023-10-14T01:22:01.025611+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:22:01.025611+0000","last_undegraded":"2023-10-14T01:22:01.025611+0000","last_fullsized":"2023-10-14T01:22:01.025611+0000","mapping_epoch":15,"log_start":"94'510134","ondisk_log_start":"94'510134","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7258554368,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":514811,"num_read_kb":2028464,"num_write":515002,"num_write_kb":2060008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"95'10684","reported_seq":14667,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:57.081071+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:21:57.081071+0000","last_peered":"2023-10-14T01:21:57.081071+0000","last_clean":"2023-10-14T01:21:57.081071+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:21:57.081071+0000","last_undegraded":"2023-10-14T01:21:57.081071+0000","last_fullsized":"2023-10-14T01:21:57.081071+0000","mapping_epoch":12,"log_start":"47'600","ondisk_log_start":"47'600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"95'528446","reported_seq":1049582,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:02.232919+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:22:02.232919+0000","last_peered":"2023-10-14T01:22:02.232919+0000","last_clean":"2023-10-14T01:22:02.232919+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:22:02.232919+0000","last_undegraded":"2023-10-14T01:22:02.232919+0000","last_fullsized":"2023-10-14T01:22:02.232919+0000","mapping_epoch":15,"log_start":"94'518428","ondisk_log_start":"94'518428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7392313344,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":523177,"num_read_kb":2061224,"num_write":523177,"num_write_kb":2092708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4095,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:57.010078+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:21:57.010078+0000","last_peered":"2023-10-14T01:21:57.010078+0000","last_clean":"2023-10-14T01:21:57.010078+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:21:57.010078+0000","last_undegraded":"2023-10-14T01:21:57.010078+0000","last_fullsized":"2023-10-14T01:21:57.010078+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"95'531777","reported_seq":1056650,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:01.081888+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:22:01.081888+0000","last_peered":"2023-10-14T01:22:01.081888+0000","last_clean":"2023-10-14T01:22:01.081888+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:22:01.081888+0000","last_undegraded":"2023-10-14T01:22:01.081888+0000","last_fullsized":"2023-10-14T01:22:01.081888+0000","mapping_epoch":15,"log_start":"94'521727","ondisk_log_start":"94'521727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7417454592,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":526932,"num_read_kb":2076780,"num_write":526494,"num_write_kb":2105976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1485,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:51.960499+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:21:51.960499+0000","last_peered":"2023-10-14T01:21:51.960499+0000","last_clean":"2023-10-14T01:21:51.960499+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:21:51.960499+0000","last_undegraded":"2023-10-14T01:21:51.960499+0000","last_fullsized":"2023-10-14T01:21:51.960499+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"95'526755","reported_seq":1049521,"reported_epoch":95,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:02.233011+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:22:02.233011+0000","last_peered":"2023-10-14T01:22:02.233011+0000","last_clean":"2023-10-14T01:22:02.233011+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:22:02.233011+0000","last_undegraded":"2023-10-14T01:22:02.233011+0000","last_fullsized":"2023-10-14T01:22:02.233011+0000","mapping_epoch":15,"log_start":"94'516734","ondisk_log_start":"94'516734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1168,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 176s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7396950035,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":524902,"num_read_kb":2068172,"num_write":523379,"num_write_kb":2093510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"95'14277","reported_seq":19457,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:22:02.130533+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:22:02.130533+0000","last_peered":"2023-10-14T01:22:02.130533+0000","last_clean":"2023-10-14T01:22:02.130533+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:22:02.130533+0000","last_undegraded":"2023-10-14T01:22:02.130533+0000","last_fullsized":"2023-10-14T01:22:02.130533+0000","mapping_epoch":12,"log_start":"81'4200","ondisk_log_start":"81'4200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":271,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:20:57.270008+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:20:57.270008+0000","last_peered":"2023-10-14T01:20:57.270008+0000","last_clean":"2023-10-14T01:20:57.270008+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:20:57.270008+0000","last_undegraded":"2023-10-14T01:20:57.270008+0000","last_fullsized":"2023-10-14T01:20:57.270008+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29465272339,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2089822,"num_read_kb":8234640,"num_write":2088052,"num_write_kb":8352202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17103486976,"data_stored":20171513894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40130,"ondisk_log_size":40130,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47749,"ondisk_log_size":47749,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739300,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9883740,"kb_used_data":8350036,"kb_used_omap":1270,"kb_used_meta":1532425,"kb_avail":84488100,"statfs":{"total":96636764160,"available":86515814400,"internally_reserved":0,"allocated":8550436864,"data_stored":10083923699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1569203898},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.40300000000000002,"15min":0.35399999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.33100000000000002,"15min":0.36199999999999999},"min":{"1min":0.218,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.40699999999999997,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.57599999999999996,"15min":0.48499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.63200000000000001,"15min":0.53300000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.8449999999999998,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.44800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739301,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5916052,"kb_used_data":4187788,"kb_used_omap":512,"kb_used_meta":1727743,"kb_avail":88455788,"statfs":{"total":96636764160,"available":90578726912,"internally_reserved":0,"allocated":4288294912,"data_stored":5055361650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1769209285},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,3,4,24],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:17 2023","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.53800000000000003,"15min":0.48099999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.152},"max":{"1min":0.70199999999999996,"5min":7.2809999999999997,"15min":7.2809999999999997},"last":4.9180000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.21099999999999999,"5min":0.187,"15min":0.17399999999999999},"max":{"1min":0.72699999999999998,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.215}]},{"osd":2,"last update":"Sat Oct 14 01:21:17 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.52200000000000002,"15min":0.42699999999999999},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.193},"max":{"1min":1.3999999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.186},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57099999999999995,"15min":0.47299999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.249},"max":{"1min":1.4690000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5886436,"kb_used_data":4167004,"kb_used_omap":504,"kb_used_meta":1718919,"kb_avail":88485404,"statfs":{"total":96636764160,"available":90609053696,"internally_reserved":0,"allocated":4267012096,"data_stored":5032529573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1760173894},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.187,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.48599999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.60999999999999999,"15min":0.53600000000000003},"min":{"1min":0.20899999999999999,"5min":0.19600000000000001,"15min":0.17399999999999999},"max":{"1min":1.954,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.246}]},{"osd":2,"last update":"Sat Oct 14 01:20:56 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":1.012,"15min":0.73799999999999999},"min":{"1min":0.182,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":1.8460000000000001,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.85399999999999998,"15min":0.57299999999999995},"min":{"1min":0.23300000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.47099999999999997,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.28999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4265967616,"data_stored":5032153088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4286689280,"data_stored":5054517267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8550830080,"data_stored":10084843539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:22:06.413 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:06.413 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:22:06.868 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:22:06.898 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:09.021 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:09.014+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:22:26.900 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:27.180 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:22:27.178+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:22:27.437 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:22:27.438 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:22:27.467 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2427,"stamp":"2023-10-14T01:22:26.478990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29503782732,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2121742,"num_read_kb":8291318,"num_write":2098245,"num_write_kb":8391991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87998,"ondisk_log_size":87998,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21986124,"kb_used_data":16782708,"kb_used_omap":2287,"kb_used_meta":5201104,"kb_avail":261129396,"statfs":{"total":289910292480,"available":267396501504,"internally_reserved":0,"allocated":17185492992,"data_stored":20251550452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":5325931461},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,12,13,18,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21499904,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5422,"num_read_kb":21596,"num_write":5324,"num_write_kb":21296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002047"},"pg_stats":[{"pgid":"2.7","version":"96'10850","reported_seq":14959,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:21.465037+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:22:21.465037+0000","last_peered":"2023-10-14T01:22:21.465037+0000","last_clean":"2023-10-14T01:22:21.465037+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:22:21.465037+0000","last_undegraded":"2023-10-14T01:22:21.465037+0000","last_fullsized":"2023-10-14T01:22:21.465037+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7305","reported_seq":11001,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:21.452010+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:22:21.452010+0000","last_peered":"2023-10-14T01:22:21.452010+0000","last_clean":"2023-10-14T01:22:21.452010+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:22:21.452010+0000","last_undegraded":"2023-10-14T01:22:21.452010+0000","last_fullsized":"2023-10-14T01:22:21.452010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7305,"log_dups_size":0,"ondisk_log_size":7305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":184,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:07.279129+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:22:07.279129+0000","last_peered":"2023-10-14T01:22:07.279129+0000","last_clean":"2023-10-14T01:22:07.279129+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:22:07.279129+0000","last_undegraded":"2023-10-14T01:22:07.279129+0000","last_fullsized":"2023-10-14T01:22:07.279129+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'14420","reported_seq":20264,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:21.475984+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:22:21.475984+0000","last_peered":"2023-10-14T01:22:21.475984+0000","last_clean":"2023-10-14T01:22:21.475984+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:22:21.475984+0000","last_undegraded":"2023-10-14T01:22:21.475984+0000","last_fullsized":"2023-10-14T01:22:21.475984+0000","mapping_epoch":12,"log_start":"81'4400","ondisk_log_start":"81'4400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'522728","reported_seq":1038076,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:26.243962+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:22:26.243962+0000","last_peered":"2023-10-14T01:22:26.243962+0000","last_clean":"2023-10-14T01:22:26.243962+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:22:26.243962+0000","last_undegraded":"2023-10-14T01:22:26.243962+0000","last_fullsized":"2023-10-14T01:22:26.243962+0000","mapping_epoch":15,"log_start":"94'512634","ondisk_log_start":"94'512634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7268913152,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":517354,"num_read_kb":2038580,"num_write":517555,"num_write_kb":2070220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'10744","reported_seq":14744,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:24.161246+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:22:24.161246+0000","last_peered":"2023-10-14T01:22:24.161246+0000","last_clean":"2023-10-14T01:22:24.161246+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:22:24.161246+0000","last_undegraded":"2023-10-14T01:22:24.161246+0000","last_fullsized":"2023-10-14T01:22:24.161246+0000","mapping_epoch":12,"log_start":"49'700","ondisk_log_start":"49'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'530577","reported_seq":1053856,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:22.348699+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:22:22.348699+0000","last_peered":"2023-10-14T01:22:22.348699+0000","last_clean":"2023-10-14T01:22:22.348699+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:22:22.348699+0000","last_undegraded":"2023-10-14T01:22:22.348699+0000","last_fullsized":"2023-10-14T01:22:22.348699+0000","mapping_epoch":15,"log_start":"94'520528","ondisk_log_start":"94'520528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7400902656,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":525318,"num_read_kb":2069748,"num_write":525308,"num_write_kb":2101232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4111,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:22.613404+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:22:22.613404+0000","last_peered":"2023-10-14T01:22:22.613404+0000","last_clean":"2023-10-14T01:22:22.613404+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:22:22.613404+0000","last_undegraded":"2023-10-14T01:22:22.613404+0000","last_fullsized":"2023-10-14T01:22:22.613404+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'534374","reported_seq":1061897,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:26.218456+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:22:26.218456+0000","last_peered":"2023-10-14T01:22:26.218456+0000","last_clean":"2023-10-14T01:22:26.218456+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:22:26.218456+0000","last_undegraded":"2023-10-14T01:22:26.218456+0000","last_fullsized":"2023-10-14T01:22:26.218456+0000","mapping_epoch":15,"log_start":"94'524327","ondisk_log_start":"94'524327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7427928064,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":529581,"num_read_kb":2087332,"num_write":529091,"num_write_kb":2116364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1491,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:17.495287+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:22:17.495287+0000","last_peered":"2023-10-14T01:22:17.495287+0000","last_clean":"2023-10-14T01:22:17.495287+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:22:17.495287+0000","last_undegraded":"2023-10-14T01:22:17.495287+0000","last_fullsized":"2023-10-14T01:22:17.495287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'528949","reported_seq":1053879,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:22.360250+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:22:22.360250+0000","last_peered":"2023-10-14T01:22:22.360250+0000","last_clean":"2023-10-14T01:22:22.360250+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:22:22.360250+0000","last_undegraded":"2023-10-14T01:22:22.360250+0000","last_fullsized":"2023-10-14T01:22:22.360250+0000","mapping_epoch":15,"log_start":"94'518934","ondisk_log_start":"94'518934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1303,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 196s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7405445139,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":527052,"num_read_kb":2076744,"num_write":525481,"num_write_kb":2101918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'14333","reported_seq":19527,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:21.503976+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:22:21.503976+0000","last_peered":"2023-10-14T01:22:21.503976+0000","last_clean":"2023-10-14T01:22:21.503976+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:22:21.503976+0000","last_undegraded":"2023-10-14T01:22:21.503976+0000","last_fullsized":"2023-10-14T01:22:21.503976+0000","mapping_epoch":12,"log_start":"81'4300","ondisk_log_start":"81'4300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:11.840830+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:21:11.840830+0000","last_peered":"2023-10-14T01:21:11.840830+0000","last_clean":"2023-10-14T01:21:11.840830+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:21:11.840830+0000","last_undegraded":"2023-10-14T01:21:11.840830+0000","last_fullsized":"2023-10-14T01:21:11.840830+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29503189011,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2099305,"num_read_kb":8272404,"num_write":2097435,"num_write_kb":8389734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17184280576,"data_stored":20252307494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40205,"ondisk_log_size":40205,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47694,"ondisk_log_size":47694,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739305,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10034488,"kb_used_data":8391216,"kb_used_omap":1270,"kb_used_meta":1641993,"kb_avail":84337352,"statfs":{"total":96636764160,"available":86361448448,"internally_reserved":0,"allocated":8592605184,"data_stored":10126087489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1681401530},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.40300000000000002,"15min":0.35399999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":5.1449999999999996,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.29599999999999999,"5min":0.33100000000000002,"15min":0.36199999999999999},"min":{"1min":0.218,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.40699999999999997,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.26500000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.57599999999999996,"15min":0.48499999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":7.1859999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.877,"5min":0.63200000000000001,"15min":0.53300000000000003},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.8449999999999998,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.52900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739305,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5986476,"kb_used_data":4204068,"kb_used_omap":512,"kb_used_meta":1781887,"kb_avail":88385364,"statfs":{"total":96636764160,"available":90506612736,"internally_reserved":0,"allocated":4304965632,"data_stored":5072027840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1824652741},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,5,11,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.158,"5min":0.158,"15min":0.152},"max":{"1min":6.0019999999999998,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.58899999999999997,"15min":0.54100000000000004},"min":{"1min":0.215,"5min":0.19600000000000001,"15min":0.17599999999999999},"max":{"1min":4.9199999999999999,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57699999999999996,"15min":0.44600000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.806,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.49399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.843,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.36499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5965160,"kb_used_data":4187424,"kb_used_omap":504,"kb_used_meta":1777223,"kb_avail":88406680,"statfs":{"total":96636764160,"available":90528440320,"internally_reserved":0,"allocated":4287922176,"data_stored":5053435123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1819877190},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.20599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.38300000000000001,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.31},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.872,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Oct 14 01:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.77900000000000003,"15min":0.72399999999999998},"min":{"1min":0.21199999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.6120000000000001,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.70699999999999996,"15min":0.58299999999999996},"min":{"1min":0.26000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.5419999999999998,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4287066112,"data_stored":5053251584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4304035840,"data_stored":5071863827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8593178624,"data_stored":10127192083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:22:27.469 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:27.469 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:22:44.410 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:44.406+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:22:44.413 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:44.410+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:22:44.413 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:44.410+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:22:44.414 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:22:44.410+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:22:47.470 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:22:47.914 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:22:47.915 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:22:47.948 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2437,"stamp":"2023-10-14T01:22:46.486687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29537574732,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2130284,"num_read_kb":8325282,"num_write":2106565,"num_write_kb":8425271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88292,"ondisk_log_size":88292,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22225396,"kb_used_data":16851228,"kb_used_omap":2287,"kb_used_meta":5371856,"kb_avail":260890124,"statfs":{"total":289910292480,"available":267151486976,"internally_reserved":0,"allocated":17255657472,"data_stored":20321714932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":5500781509},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,2,5,3,5,7,11,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21032960,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5230,"num_read_kb":20804,"num_write":5178,"num_write_kb":20712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006294"},"pg_stats":[{"pgid":"2.7","version":"96'10889","reported_seq":15010,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:40.688003+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:22:40.688003+0000","last_peered":"2023-10-14T01:22:40.688003+0000","last_clean":"2023-10-14T01:22:40.688003+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:22:40.688003+0000","last_undegraded":"2023-10-14T01:22:40.688003+0000","last_fullsized":"2023-10-14T01:22:40.688003+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7331","reported_seq":11035,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:40.678391+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:22:40.678391+0000","last_peered":"2023-10-14T01:22:40.678391+0000","last_clean":"2023-10-14T01:22:40.678391+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:22:40.678391+0000","last_undegraded":"2023-10-14T01:22:40.678391+0000","last_fullsized":"2023-10-14T01:22:40.678391+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7331,"log_dups_size":0,"ondisk_log_size":7331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":184,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:07.279129+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:22:07.279129+0000","last_peered":"2023-10-14T01:22:07.279129+0000","last_clean":"2023-10-14T01:22:07.279129+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:22:07.279129+0000","last_undegraded":"2023-10-14T01:22:07.279129+0000","last_fullsized":"2023-10-14T01:22:07.279129+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'14472","reported_seq":20332,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:40.689552+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:22:40.689552+0000","last_peered":"2023-10-14T01:22:40.689552+0000","last_clean":"2023-10-14T01:22:40.689552+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:22:40.689552+0000","last_undegraded":"2023-10-14T01:22:40.689552+0000","last_fullsized":"2023-10-14T01:22:40.689552+0000","mapping_epoch":12,"log_start":"81'4400","ondisk_log_start":"81'4400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'524831","reported_seq":1042354,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:46.320572+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:22:46.320572+0000","last_peered":"2023-10-14T01:22:46.320572+0000","last_clean":"2023-10-14T01:22:46.320572+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:22:46.320572+0000","last_undegraded":"2023-10-14T01:22:46.320572+0000","last_fullsized":"2023-10-14T01:22:46.320572+0000","mapping_epoch":15,"log_start":"95'514734","ondisk_log_start":"95'514734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7277461504,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":519531,"num_read_kb":2047228,"num_write":519658,"num_write_kb":2078632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'10790","reported_seq":14802,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:44.541452+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:22:44.541452+0000","last_peered":"2023-10-14T01:22:44.541452+0000","last_clean":"2023-10-14T01:22:44.541452+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:22:44.541452+0000","last_undegraded":"2023-10-14T01:22:44.541452+0000","last_fullsized":"2023-10-14T01:22:44.541452+0000","mapping_epoch":12,"log_start":"49'700","ondisk_log_start":"49'700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'532658","reported_seq":1058066,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:42.488488+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:22:42.488488+0000","last_peered":"2023-10-14T01:22:42.488488+0000","last_clean":"2023-10-14T01:22:42.488488+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:22:42.488488+0000","last_undegraded":"2023-10-14T01:22:42.488488+0000","last_fullsized":"2023-10-14T01:22:42.488488+0000","mapping_epoch":15,"log_start":"95'522628","ondisk_log_start":"95'522628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7409352704,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":527447,"num_read_kb":2078216,"num_write":527389,"num_write_kb":2109556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4123,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:43.091556+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:22:43.091556+0000","last_peered":"2023-10-14T01:22:43.091556+0000","last_clean":"2023-10-14T01:22:43.091556+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:22:43.091556+0000","last_undegraded":"2023-10-14T01:22:43.091556+0000","last_fullsized":"2023-10-14T01:22:43.091556+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'536421","reported_seq":1066039,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:46.319586+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:22:46.319586+0000","last_peered":"2023-10-14T01:22:46.319586+0000","last_clean":"2023-10-14T01:22:46.319586+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:22:46.319586+0000","last_undegraded":"2023-10-14T01:22:46.319586+0000","last_fullsized":"2023-10-14T01:22:46.319586+0000","mapping_epoch":15,"log_start":"95'526327","ondisk_log_start":"95'526327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7436251136,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":531676,"num_read_kb":2095672,"num_write":531138,"num_write_kb":2124552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1495,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:38.011945+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:22:38.011945+0000","last_peered":"2023-10-14T01:22:38.011945+0000","last_clean":"2023-10-14T01:22:38.011945+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:22:38.011945+0000","last_undegraded":"2023-10-14T01:22:38.011945+0000","last_fullsized":"2023-10-14T01:22:38.011945+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'531097","reported_seq":1058180,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:22:42.513455+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:22:42.513455+0000","last_peered":"2023-10-14T01:22:42.513455+0000","last_clean":"2023-10-14T01:22:42.513455+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:22:42.513455+0000","last_undegraded":"2023-10-14T01:22:42.513455+0000","last_fullsized":"2023-10-14T01:22:42.513455+0000","mapping_epoch":15,"log_start":"95'521034","ondisk_log_start":"95'521034","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1390,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 216s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7413915667,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":529193,"num_read_kb":2085252,"num_write":527570,"num_write_kb":2110274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'14385","reported_seq":19595,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:40.704935+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:22:40.704935+0000","last_peered":"2023-10-14T01:22:40.704935+0000","last_clean":"2023-10-14T01:22:40.704935+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:22:40.704935+0000","last_undegraded":"2023-10-14T01:22:40.704935+0000","last_fullsized":"2023-10-14T01:22:40.704935+0000","mapping_epoch":12,"log_start":"81'4300","ondisk_log_start":"81'4300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:11.840830+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:21:11.840830+0000","last_peered":"2023-10-14T01:21:11.840830+0000","last_clean":"2023-10-14T01:21:11.840830+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:21:11.840830+0000","last_undegraded":"2023-10-14T01:21:11.840830+0000","last_fullsized":"2023-10-14T01:21:11.840830+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29536981011,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2107847,"num_read_kb":8306368,"num_write":2105755,"num_write_kb":8423014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17252438016,"data_stored":20320464934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40284,"ondisk_log_size":40284,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47909,"ondisk_log_size":47909,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739309,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10141304,"kb_used_data":8425840,"kb_used_omap":1270,"kb_used_meta":1714185,"kb_avail":84230536,"statfs":{"total":96636764160,"available":86252068864,"internally_reserved":0,"allocated":8628060160,"data_stored":10161542465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1755326138},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.435,"15min":0.36499999999999999},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":3.1099999999999999,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.33600000000000002,"15min":0.35599999999999998},"min":{"1min":0.17299999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.56299999999999994,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.34399999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":0.753,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.314},{"interface":"front","average":{"1min":0.437,"5min":0.56100000000000005,"15min":0.52100000000000002},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78400000000000003,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.376}]}]},{"osd":1,"up_from":8,"seq":34359739309,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044100,"kb_used_data":4221436,"kb_used_omap":512,"kb_used_meta":1822143,"kb_avail":88327740,"statfs":{"total":96636764160,"available":90447605760,"internally_reserved":0,"allocated":4322750464,"data_stored":5089812672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1865874885},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,0,3,1,7,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.158,"5min":0.158,"15min":0.152},"max":{"1min":6.0019999999999998,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.16900000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.58899999999999997,"15min":0.54100000000000004},"min":{"1min":0.215,"5min":0.19600000000000001,"15min":0.17599999999999999},"max":{"1min":4.9199999999999999,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.184}]},{"osd":2,"last update":"Sat Oct 14 01:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57699999999999996,"15min":0.44600000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.806,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.49399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.843,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.23899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739312,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6039992,"kb_used_data":4203952,"kb_used_omap":504,"kb_used_meta":1835527,"kb_avail":88331848,"statfs":{"total":96636764160,"available":90451812352,"internally_reserved":0,"allocated":4304846848,"data_stored":5070359795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1879580486},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.45100000000000001,"15min":0.46000000000000002},"min":{"1min":0.20599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.38300000000000001,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.20599999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":0.872,"5min":6.8369999999999997,"15min":6.8369999999999997},"last":0.23899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.77900000000000003,"15min":0.72399999999999998},"min":{"1min":0.21199999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":4.6120000000000001,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.70699999999999996,"15min":0.58299999999999996},"min":{"1min":0.26000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.5419999999999998,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4304064512,"data_stored":5070249984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4321120256,"data_stored":5088948243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8627253248,"data_stored":10161266707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:22:47.950 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:22:47.950 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:07.952 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:08.454 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:23:08.455 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:23:08.486 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2447,"stamp":"2023-10-14T01:23:06.490183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29571325772,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2138728,"num_read_kb":8358906,"num_write":2114880,"num_write_kb":8458531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88095,"ondisk_log_size":88095,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22495480,"kb_used_data":16917216,"kb_used_omap":2287,"kb_used_meta":5575952,"kb_avail":260620040,"statfs":{"total":289910292480,"available":266874920960,"internally_reserved":0,"allocated":17323229184,"data_stored":20389286644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":5709775813},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,5,8,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21217280,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5301,"num_read_kb":21092,"num_write":5224,"num_write_kb":20896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002144"},"pg_stats":[{"pgid":"2.7","version":"96'10929","reported_seq":15062,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:02.605212+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:23:02.605212+0000","last_peered":"2023-10-14T01:23:02.605212+0000","last_clean":"2023-10-14T01:23:02.605212+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:23:02.605212+0000","last_undegraded":"2023-10-14T01:23:02.605212+0000","last_fullsized":"2023-10-14T01:23:02.605212+0000","mapping_epoch":12,"log_start":"51'900","ondisk_log_start":"51'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7359","reported_seq":11071,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:02.588655+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:23:02.588655+0000","last_peered":"2023-10-14T01:23:02.588655+0000","last_clean":"2023-10-14T01:23:02.588655+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:23:02.588655+0000","last_undegraded":"2023-10-14T01:23:02.588655+0000","last_fullsized":"2023-10-14T01:23:02.588655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":184,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:07.279129+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:22:07.279129+0000","last_peered":"2023-10-14T01:22:07.279129+0000","last_clean":"2023-10-14T01:22:07.279129+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:22:07.279129+0000","last_undegraded":"2023-10-14T01:22:07.279129+0000","last_fullsized":"2023-10-14T01:22:07.279129+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'14525","reported_seq":20400,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:02.614461+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:23:02.614461+0000","last_peered":"2023-10-14T01:23:02.614461+0000","last_clean":"2023-10-14T01:23:02.614461+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:23:02.614461+0000","last_undegraded":"2023-10-14T01:23:02.614461+0000","last_fullsized":"2023-10-14T01:23:02.614461+0000","mapping_epoch":12,"log_start":"81'4500","ondisk_log_start":"81'4500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'526877","reported_seq":1046427,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:06.413961+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:23:06.413961+0000","last_peered":"2023-10-14T01:23:06.413961+0000","last_clean":"2023-10-14T01:23:06.413961+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:23:06.413961+0000","last_undegraded":"2023-10-14T01:23:06.413961+0000","last_fullsized":"2023-10-14T01:23:06.413961+0000","mapping_epoch":15,"log_start":"95'516834","ondisk_log_start":"95'516834","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7285772288,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":521558,"num_read_kb":2055304,"num_write":521704,"num_write_kb":2086816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'10833","reported_seq":14857,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:05.443791+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:23:05.443791+0000","last_peered":"2023-10-14T01:23:05.443791+0000","last_clean":"2023-10-14T01:23:05.443791+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:23:05.443791+0000","last_undegraded":"2023-10-14T01:23:05.443791+0000","last_fullsized":"2023-10-14T01:23:05.443791+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"96'534700","reported_seq":1062198,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:02.611479+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:23:02.611479+0000","last_peered":"2023-10-14T01:23:02.611479+0000","last_clean":"2023-10-14T01:23:02.611479+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:23:02.611479+0000","last_undegraded":"2023-10-14T01:23:02.611479+0000","last_fullsized":"2023-10-14T01:23:02.611479+0000","mapping_epoch":15,"log_start":"95'524628","ondisk_log_start":"95'524628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7417638912,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":529541,"num_read_kb":2086548,"num_write":529431,"num_write_kb":2117724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4135,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:03.600598+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:23:03.600598+0000","last_peered":"2023-10-14T01:23:03.600598+0000","last_clean":"2023-10-14T01:23:03.600598+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:23:03.600598+0000","last_undegraded":"2023-10-14T01:23:03.600598+0000","last_fullsized":"2023-10-14T01:23:03.600598+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'538512","reported_seq":1070388,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:06.419460+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:23:06.419460+0000","last_peered":"2023-10-14T01:23:06.419460+0000","last_clean":"2023-10-14T01:23:06.419460+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:23:06.419460+0000","last_undegraded":"2023-10-14T01:23:06.419460+0000","last_fullsized":"2023-10-14T01:23:06.419460+0000","mapping_epoch":15,"log_start":"95'528427","ondisk_log_start":"95'528427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7444770816,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":533937,"num_read_kb":2104668,"num_write":533229,"num_write_kb":2132916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1499,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:58.450406+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:22:58.450406+0000","last_peered":"2023-10-14T01:22:58.450406+0000","last_clean":"2023-10-14T01:22:58.450406+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:22:58.450406+0000","last_undegraded":"2023-10-14T01:22:58.450406+0000","last_fullsized":"2023-10-14T01:22:58.450406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"96'533308","reported_seq":1062462,"reported_epoch":96,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:02.619385+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:23:02.619385+0000","last_peered":"2023-10-14T01:23:02.619385+0000","last_clean":"2023-10-14T01:23:02.619385+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:23:02.619385+0000","last_undegraded":"2023-10-14T01:23:02.619385+0000","last_fullsized":"2023-10-14T01:23:02.619385+0000","mapping_epoch":15,"log_start":"95'523234","ondisk_log_start":"95'523234","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1501,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 236s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7422550035,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":531255,"num_read_kb":2093472,"num_write":529706,"num_write_kb":2118818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'14434","reported_seq":19660,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:00.195108+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:23:00.195108+0000","last_peered":"2023-10-14T01:23:00.195108+0000","last_clean":"2023-10-14T01:23:00.195108+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:23:00.195108+0000","last_undegraded":"2023-10-14T01:23:00.195108+0000","last_fullsized":"2023-10-14T01:23:00.195108+0000","mapping_epoch":12,"log_start":"81'4400","ondisk_log_start":"81'4400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T01:21:11.840830+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:21:11.840830+0000","last_peered":"2023-10-14T01:21:11.840830+0000","last_clean":"2023-10-14T01:21:11.840830+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:21:11.840830+0000","last_undegraded":"2023-10-14T01:21:11.840830+0000","last_fullsized":"2023-10-14T01:21:11.840830+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29570732051,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2116291,"num_read_kb":8339992,"num_write":2114070,"num_write_kb":8456274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17320235008,"data_stored":20388261926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40274,"ondisk_log_size":40274,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47722,"ondisk_log_size":47722,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739313,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10287940,"kb_used_data":8458620,"kb_used_omap":1270,"kb_used_meta":1828041,"kb_avail":84083900,"statfs":{"total":96636764160,"available":86101913600,"internally_reserved":0,"allocated":8661626880,"data_stored":10195109185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1871914682},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.435,"15min":0.36499999999999999},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":3.1099999999999999,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.33600000000000002,"15min":0.35599999999999998},"min":{"1min":0.17299999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.56299999999999994,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.432}]},{"osd":1,"last update":"Sat Oct 14 01:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":0.753,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.56100000000000005,"15min":0.52100000000000002},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78400000000000003,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.36099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739313,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6114860,"kb_used_data":4238052,"kb_used_omap":512,"kb_used_meta":1876287,"kb_avail":88256980,"statfs":{"total":96636764160,"available":90375147520,"internally_reserved":0,"allocated":4339765248,"data_stored":5106827456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1921318341},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,5,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:21 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.158,"5min":0.158,"15min":0.152},"max":{"1min":6.0019999999999998,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.58899999999999997,"15min":0.54100000000000004},"min":{"1min":0.215,"5min":0.19600000000000001,"15min":0.17599999999999999},"max":{"1min":4.9199999999999999,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.19700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57699999999999996,"15min":0.44600000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.806,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.49399999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.843,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739316,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6092680,"kb_used_data":4220544,"kb_used_omap":504,"kb_used_meta":1871623,"kb_avail":88279160,"statfs":{"total":96636764160,"available":90397859840,"internally_reserved":0,"allocated":4321837056,"data_stored":5087350003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1916542790},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.39800000000000002,"15min":0.45700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.48699999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":3.1600000000000001,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":0.247}]},{"osd":2,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.73599999999999999,"15min":0.72799999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":0.85499999999999998,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.71299999999999997,"15min":0.58199999999999996},"min":{"1min":0.25700000000000001,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.41699999999999998,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4321013760,"data_stored":5087199232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4338225152,"data_stored":5106053139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8660996096,"data_stored":10195009555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:23:08.488 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:08.489 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:28.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:28.959 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:23:28.959 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:23:28.986 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2459,"stamp":"2023-10-14T01:23:28.493780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29608890188,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2148595,"num_read_kb":8398122,"num_write":2124148,"num_write_kb":8495603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88188,"ondisk_log_size":88188,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22740252,"kb_used_data":16987712,"kb_used_omap":2287,"kb_used_meta":5750224,"kb_avail":260375268,"statfs":{"total":289910292480,"available":266624274432,"internally_reserved":0,"allocated":17395417088,"data_stored":20461469975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":5888230341},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,10,6,5,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16687104,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4273,"num_read_kb":16964,"num_write":4115,"num_write_kb":16460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001603"},"pg_stats":[{"pgid":"2.7","version":"96'10968","reported_seq":15118,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.453701+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:23:25.453701+0000","last_peered":"2023-10-14T01:23:25.453701+0000","last_clean":"2023-10-14T01:23:25.453701+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:23:25.453701+0000","last_undegraded":"2023-10-14T01:23:25.453701+0000","last_fullsized":"2023-10-14T01:23:25.453701+0000","mapping_epoch":12,"log_start":"51'900","ondisk_log_start":"51'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"96'7385","reported_seq":11109,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.442333+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:23:25.442333+0000","last_peered":"2023-10-14T01:23:25.442333+0000","last_clean":"2023-10-14T01:23:25.442333+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:23:25.442333+0000","last_undegraded":"2023-10-14T01:23:25.442333+0000","last_fullsized":"2023-10-14T01:23:25.442333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":185,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:22.367706+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:22:22.367706+0000","last_peered":"2023-10-14T01:22:22.367706+0000","last_clean":"2023-10-14T01:22:22.367706+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:22:22.367706+0000","last_undegraded":"2023-10-14T01:22:22.367706+0000","last_fullsized":"2023-10-14T01:22:22.367706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"96'14577","reported_seq":20474,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.455266+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:23:25.455266+0000","last_peered":"2023-10-14T01:23:25.455266+0000","last_clean":"2023-10-14T01:23:25.455266+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:23:25.455266+0000","last_undegraded":"2023-10-14T01:23:25.455266+0000","last_fullsized":"2023-10-14T01:23:25.455266+0000","mapping_epoch":12,"log_start":"81'4500","ondisk_log_start":"81'4500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"96'528984","reported_seq":1050716,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:26.509714+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:23:26.509714+0000","last_peered":"2023-10-14T01:23:26.509714+0000","last_clean":"2023-10-14T01:23:26.509714+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:23:26.509714+0000","last_undegraded":"2023-10-14T01:23:26.509714+0000","last_fullsized":"2023-10-14T01:23:26.509714+0000","mapping_epoch":15,"log_start":"95'518934","ondisk_log_start":"95'518934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7294316544,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":523738,"num_read_kb":2063972,"num_write":523811,"num_write_kb":2095244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"96'10872","reported_seq":14908,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.897313+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:23:25.897313+0000","last_peered":"2023-10-14T01:23:25.897313+0000","last_clean":"2023-10-14T01:23:25.897313+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:23:25.897313+0000","last_undegraded":"2023-10-14T01:23:25.897313+0000","last_fullsized":"2023-10-14T01:23:25.897313+0000","mapping_epoch":12,"log_start":"49'800","ondisk_log_start":"49'800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'537167","reported_seq":1067434,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:27.727403+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:23:27.727403+0000","last_peered":"2023-10-14T01:23:27.727403+0000","last_clean":"2023-10-14T01:23:27.727403+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:23:27.727403+0000","last_undegraded":"2023-10-14T01:23:27.727403+0000","last_fullsized":"2023-10-14T01:23:27.727403+0000","mapping_epoch":15,"log_start":"95'527128","ondisk_log_start":"95'527128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7427641344,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":532304,"num_read_kb":2097552,"num_write":531898,"num_write_kb":2127592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4147,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:24.050106+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:23:24.050106+0000","last_peered":"2023-10-14T01:23:24.050106+0000","last_clean":"2023-10-14T01:23:24.050106+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:23:24.050106+0000","last_undegraded":"2023-10-14T01:23:24.050106+0000","last_fullsized":"2023-10-14T01:23:24.050106+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"96'540630","reported_seq":1074743,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:26.496037+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:23:26.496037+0000","last_peered":"2023-10-14T01:23:26.496037+0000","last_clean":"2023-10-14T01:23:26.496037+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:23:26.496037+0000","last_undegraded":"2023-10-14T01:23:26.496037+0000","last_fullsized":"2023-10-14T01:23:26.496037+0000","mapping_epoch":15,"log_start":"95'530627","ondisk_log_start":"95'530627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7453331456,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":536171,"num_read_kb":2113556,"num_write":535347,"num_write_kb":2141388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1504,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:18.933687+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:23:18.933687+0000","last_peered":"2023-10-14T01:23:18.933687+0000","last_clean":"2023-10-14T01:23:18.933687+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:23:18.933687+0000","last_undegraded":"2023-10-14T01:23:18.933687+0000","last_fullsized":"2023-10-14T01:23:18.933687+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'535997","reported_seq":1067858,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:27.733334+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:23:27.733334+0000","last_peered":"2023-10-14T01:23:27.733334+0000","last_clean":"2023-10-14T01:23:27.733334+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:23:27.733334+0000","last_undegraded":"2023-10-14T01:23:27.733334+0000","last_fullsized":"2023-10-14T01:23:27.733334+0000","mapping_epoch":15,"log_start":"95'525934","ondisk_log_start":"95'525934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1663,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 261s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7433007123,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":533945,"num_read_kb":2104128,"num_write":532282,"num_write_kb":2129122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"96'14490","reported_seq":19737,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:25.460481+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:23:25.460481+0000","last_peered":"2023-10-14T01:23:25.460481+0000","last_clean":"2023-10-14T01:23:25.460481+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:23:25.460481+0000","last_undegraded":"2023-10-14T01:23:25.460481+0000","last_fullsized":"2023-10-14T01:23:25.460481+0000","mapping_epoch":12,"log_start":"81'4400","ondisk_log_start":"81'4400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:12.687920+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:23:12.687920+0000","last_peered":"2023-10-14T01:23:12.687920+0000","last_clean":"2023-10-14T01:23:12.687920+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:23:12.687920+0000","last_undegraded":"2023-10-14T01:23:12.687920+0000","last_fullsized":"2023-10-14T01:23:12.687920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29608296467,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2126158,"num_read_kb":8379208,"num_write":2123338,"num_write_kb":8493346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17392156672,"data_stored":20460183590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40155,"ondisk_log_size":40155,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47934,"ondisk_log_size":47934,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739317,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10411400,"kb_used_data":8491840,"kb_used_omap":1270,"kb_used_meta":1918281,"kb_avail":83960440,"statfs":{"total":96636764160,"available":85975490560,"internally_reserved":0,"allocated":8695644160,"data_stored":10229126465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":1964320442},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.435,"15min":0.36499999999999999},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":3.1099999999999999,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.29199999999999998,"5min":0.33600000000000002,"15min":0.35599999999999998},"min":{"1min":0.17299999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.56299999999999994,"5min":2.9449999999999998,"15min":5.1109999999999998},"last":0.48299999999999998}]},{"osd":1,"last update":"Sat Oct 14 01:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":0.753,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.56100000000000005,"15min":0.52100000000000002},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.78400000000000003,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.33500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739318,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6183104,"kb_used_data":4258356,"kb_used_omap":512,"kb_used_meta":1924223,"kb_avail":88188736,"statfs":{"total":96636764160,"available":90305265664,"internally_reserved":0,"allocated":4360556544,"data_stored":5127614179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":1970404805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,5,2,9,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.55500000000000005,"15min":0.51100000000000001},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.152},"max":{"1min":0.99399999999999999,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.193},{"interface":"front","average":{"1min":0.317,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":0.629,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.33100000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.49099999999999999,"15min":0.44800000000000001},"min":{"1min":0.187,"5min":0.186,"15min":0.186},"max":{"1min":1.034,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.441},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.54200000000000004,"15min":0.496},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.1040000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739320,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6145748,"kb_used_data":4237516,"kb_used_omap":504,"kb_used_meta":1907719,"kb_avail":88226092,"statfs":{"total":96636764160,"available":90343518208,"internally_reserved":0,"allocated":4339216384,"data_stored":5104729331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1953505094},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.39800000000000002,"15min":0.45700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.48699999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":3.1600000000000001,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":0.246}]},{"osd":2,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.73599999999999999,"15min":0.72799999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":0.85499999999999998,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.71299999999999997,"15min":0.58199999999999996},"min":{"1min":0.25700000000000001,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.41699999999999998,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.27000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4338335744,"data_stored":5104521216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4358877184,"data_stored":5126705171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8694943744,"data_stored":10228957203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:23:28.987 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:28.988 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:23:48.990 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:23:49.654 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:23:49.655 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:23:49.680 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2469,"stamp":"2023-10-14T01:23:48.497040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29642284876,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2157313,"num_read_kb":8432742,"num_write":2132375,"num_write_kb":8528511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88186,"ondisk_log_size":88186,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23015696,"kb_used_data":17052216,"kb_used_omap":2287,"kb_used_meta":5961168,"kb_avail":260099824,"statfs":{"total":289910292480,"available":266342219776,"internally_reserved":0,"allocated":17461469184,"data_stored":20527512925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2341947,"internal_metadata":6104236997},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,1,2,11,7,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25182208,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6512,"num_read_kb":25880,"num_write":6200,"num_write_kb":24800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001946"},"pg_stats":[{"pgid":"2.7","version":"97'11001","reported_seq":15163,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:47.090056+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:23:47.090056+0000","last_peered":"2023-10-14T01:23:47.090056+0000","last_clean":"2023-10-14T01:23:47.090056+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:23:47.090056+0000","last_undegraded":"2023-10-14T01:23:47.090056+0000","last_fullsized":"2023-10-14T01:23:47.090056+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'7407","reported_seq":11139,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:47.078066+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:23:47.078066+0000","last_peered":"2023-10-14T01:23:47.078066+0000","last_clean":"2023-10-14T01:23:47.078066+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:23:47.078066+0000","last_undegraded":"2023-10-14T01:23:47.078066+0000","last_fullsized":"2023-10-14T01:23:47.078066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":185,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:22.367706+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:22:22.367706+0000","last_peered":"2023-10-14T01:22:22.367706+0000","last_clean":"2023-10-14T01:22:22.367706+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:22:22.367706+0000","last_undegraded":"2023-10-14T01:22:22.367706+0000","last_fullsized":"2023-10-14T01:22:22.367706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'14621","reported_seq":20534,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:47.100853+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:23:47.100853+0000","last_peered":"2023-10-14T01:23:47.100853+0000","last_clean":"2023-10-14T01:23:47.100853+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:23:47.100853+0000","last_undegraded":"2023-10-14T01:23:47.100853+0000","last_fullsized":"2023-10-14T01:23:47.100853+0000","mapping_epoch":12,"log_start":"81'4600","ondisk_log_start":"81'4600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'531029","reported_seq":1054857,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:46.642468+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:23:46.642468+0000","last_peered":"2023-10-14T01:23:46.642468+0000","last_clean":"2023-10-14T01:23:46.642468+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:23:46.642468+0000","last_undegraded":"2023-10-14T01:23:46.642468+0000","last_fullsized":"2023-10-14T01:23:46.642468+0000","mapping_epoch":15,"log_start":"96'520934","ondisk_log_start":"96'520934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7302639616,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":525837,"num_read_kb":2072304,"num_write":525856,"num_write_kb":2103424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'10914","reported_seq":14963,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:45.726078+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:23:45.726078+0000","last_peered":"2023-10-14T01:23:45.726078+0000","last_clean":"2023-10-14T01:23:45.726078+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:23:45.726078+0000","last_undegraded":"2023-10-14T01:23:45.726078+0000","last_fullsized":"2023-10-14T01:23:45.726078+0000","mapping_epoch":12,"log_start":"51'900","ondisk_log_start":"51'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'539224","reported_seq":1071657,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:47.806268+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:23:47.806268+0000","last_peered":"2023-10-14T01:23:47.806268+0000","last_clean":"2023-10-14T01:23:47.806268+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:23:47.806268+0000","last_undegraded":"2023-10-14T01:23:47.806268+0000","last_fullsized":"2023-10-14T01:23:47.806268+0000","mapping_epoch":15,"log_start":"96'529128","ondisk_log_start":"96'529128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7436009472,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":534472,"num_read_kb":2106164,"num_write":533955,"num_write_kb":2135820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4160,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:44.552325+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:23:44.552325+0000","last_peered":"2023-10-14T01:23:44.552325+0000","last_clean":"2023-10-14T01:23:44.552325+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:23:44.552325+0000","last_undegraded":"2023-10-14T01:23:44.552325+0000","last_fullsized":"2023-10-14T01:23:44.552325+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'542704","reported_seq":1079043,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:46.673175+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:23:46.673175+0000","last_peered":"2023-10-14T01:23:46.673175+0000","last_clean":"2023-10-14T01:23:46.673175+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:23:46.673175+0000","last_undegraded":"2023-10-14T01:23:46.673175+0000","last_fullsized":"2023-10-14T01:23:46.673175+0000","mapping_epoch":15,"log_start":"96'532627","ondisk_log_start":"96'532627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7461711872,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":538399,"num_read_kb":2122412,"num_write":537421,"num_write_kb":2149684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1509,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:39.431575+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:23:39.431575+0000","last_peered":"2023-10-14T01:23:39.431575+0000","last_clean":"2023-10-14T01:23:39.431575+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:23:39.431575+0000","last_undegraded":"2023-10-14T01:23:39.431575+0000","last_fullsized":"2023-10-14T01:23:39.431575+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'538134","reported_seq":1072231,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:23:47.801361+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:23:47.801361+0000","last_peered":"2023-10-14T01:23:47.801361+0000","last_clean":"2023-10-14T01:23:47.801361+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:23:47.801361+0000","last_undegraded":"2023-10-14T01:23:47.801361+0000","last_fullsized":"2023-10-14T01:23:47.801361+0000","mapping_epoch":15,"log_start":"96'528034","ondisk_log_start":"96'528034","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1789,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 281s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7441330195,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":536168,"num_read_kb":2112948,"num_write":534333,"num_write_kb":2137326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'14534","reported_seq":19797,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:47.139724+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:23:47.139724+0000","last_peered":"2023-10-14T01:23:47.139724+0000","last_clean":"2023-10-14T01:23:47.139724+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:23:47.139724+0000","last_undegraded":"2023-10-14T01:23:47.139724+0000","last_fullsized":"2023-10-14T01:23:47.139724+0000","mapping_epoch":12,"log_start":"81'4500","ondisk_log_start":"81'4500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:12.687920+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:23:12.687920+0000","last_peered":"2023-10-14T01:23:12.687920+0000","last_clean":"2023-10-14T01:23:12.687920+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:23:12.687920+0000","last_undegraded":"2023-10-14T01:23:12.687920+0000","last_fullsized":"2023-10-14T01:23:12.687920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29641691155,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2134876,"num_read_kb":8413828,"num_write":2131565,"num_write_kb":8526254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17459445760,"data_stored":20527472678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40368,"ondisk_log_size":40368,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47719,"ondisk_log_size":47719,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10564540,"kb_used_data":8524276,"kb_used_omap":1270,"kb_used_meta":2038985,"kb_avail":83807300,"statfs":{"total":96636764160,"available":85818675200,"internally_reserved":0,"allocated":8728858624,"data_stored":10262336356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1300806,"internal_metadata":2087921338},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.41999999999999998,"15min":0.36199999999999999},"min":{"1min":0.183,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.187},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.35199999999999998,"15min":0.35299999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.7529999999999999,"5min":2.9449999999999998,"15min":3.7519999999999998},"last":0.224}]},{"osd":1,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.52300000000000002,"15min":0.47799999999999998},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.5960000000000001,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.637,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":1.637}]}]},{"osd":1,"up_from":8,"seq":34359739322,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6252880,"kb_used_data":4273992,"kb_used_omap":512,"kb_used_meta":1978367,"kb_avail":88118960,"statfs":{"total":96636764160,"available":90233815040,"internally_reserved":0,"allocated":4376567808,"data_stored":5143625443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":2025848261},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,0,3,4,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.55500000000000005,"15min":0.51100000000000001},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.152},"max":{"1min":0.99399999999999999,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":0.629,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.23300000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.49099999999999999,"15min":0.44800000000000001},"min":{"1min":0.187,"5min":0.186,"15min":0.186},"max":{"1min":1.034,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.54200000000000004,"15min":0.496},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.1040000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.47699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739324,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6198276,"kb_used_data":4253948,"kb_used_omap":504,"kb_used_meta":1943815,"kb_avail":88173564,"statfs":{"total":96636764160,"available":90289729536,"internally_reserved":0,"allocated":4356042752,"data_stored":5121551126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516282,"internal_metadata":1990467398},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,1,6,2,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.39800000000000002,"15min":0.45700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.48699999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":3.1600000000000001,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.73599999999999999,"15min":0.72799999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":0.85499999999999998,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.315},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.71299999999999997,"15min":0.58199999999999996},"min":{"1min":0.25700000000000001,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.41699999999999998,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.33700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4355182592,"data_stored":5121368064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4375715840,"data_stored":5143543827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8728547328,"data_stored":10262560787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:23:49.682 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:23:49.682 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:01.853 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:01.852+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:24:01.854 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:01.852+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:24:01.869 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:01.868+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:24:09.683 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:10.135 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:24:10.136 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:24:10.162 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2479,"stamp":"2023-10-14T01:24:08.502388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29675458380,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2165910,"num_read_kb":8466994,"num_write":2140574,"num_write_kb":8561307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88234,"ondisk_log_size":88234,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22817324,"kb_used_data":17118356,"kb_used_omap":2691,"kb_used_meta":5696252,"kb_avail":260298196,"statfs":{"total":289910292480,"available":266545352704,"internally_reserved":0,"allocated":17529196544,"data_stored":20595240285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2755647,"internal_metadata":5832963009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,8,8,12,18,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24797184,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6369,"num_read_kb":25388,"num_write":6128,"num_write_kb":24512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003782"},"pg_stats":[{"pgid":"2.7","version":"97'11029","reported_seq":15203,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.735132+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:24:05.735132+0000","last_peered":"2023-10-14T01:24:05.735132+0000","last_clean":"2023-10-14T01:24:05.735132+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:24:05.735132+0000","last_undegraded":"2023-10-14T01:24:05.735132+0000","last_fullsized":"2023-10-14T01:24:05.735132+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'7426","reported_seq":11166,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.717597+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:24:05.717597+0000","last_peered":"2023-10-14T01:24:05.717597+0000","last_clean":"2023-10-14T01:24:05.717597+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:24:05.717597+0000","last_undegraded":"2023-10-14T01:24:05.717597+0000","last_fullsized":"2023-10-14T01:24:05.717597+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":185,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:22:22.367706+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:22:22.367706+0000","last_peered":"2023-10-14T01:22:22.367706+0000","last_clean":"2023-10-14T01:22:22.367706+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:22:22.367706+0000","last_undegraded":"2023-10-14T01:22:22.367706+0000","last_fullsized":"2023-10-14T01:22:22.367706+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'14658","reported_seq":20587,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.735634+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:24:05.735634+0000","last_peered":"2023-10-14T01:24:05.735634+0000","last_clean":"2023-10-14T01:24:05.735634+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:24:05.735634+0000","last_undegraded":"2023-10-14T01:24:05.735634+0000","last_fullsized":"2023-10-14T01:24:05.735634+0000","mapping_epoch":12,"log_start":"81'4600","ondisk_log_start":"81'4600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'533031","reported_seq":1059004,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.729667+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:24:06.729667+0000","last_peered":"2023-10-14T01:24:06.729667+0000","last_clean":"2023-10-14T01:24:06.729667+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:24:06.729667+0000","last_undegraded":"2023-10-14T01:24:06.729667+0000","last_fullsized":"2023-10-14T01:24:06.729667+0000","mapping_epoch":15,"log_start":"96'522934","ondisk_log_start":"96'522934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7310774272,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":527979,"num_read_kb":2080840,"num_write":527858,"num_write_kb":2111432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'10947","reported_seq":15008,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.009660+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:24:05.009660+0000","last_peered":"2023-10-14T01:24:05.009660+0000","last_clean":"2023-10-14T01:24:05.009660+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:24:05.009660+0000","last_undegraded":"2023-10-14T01:24:05.009660+0000","last_fullsized":"2023-10-14T01:24:05.009660+0000","mapping_epoch":12,"log_start":"51'900","ondisk_log_start":"51'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'541273","reported_seq":1075886,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:07.916523+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:24:07.916523+0000","last_peered":"2023-10-14T01:24:07.916523+0000","last_clean":"2023-10-14T01:24:07.916523+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:24:07.916523+0000","last_undegraded":"2023-10-14T01:24:07.916523+0000","last_fullsized":"2023-10-14T01:24:07.916523+0000","mapping_epoch":15,"log_start":"96'531228","ondisk_log_start":"96'531228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7444291584,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":536650,"num_read_kb":2114836,"num_write":536004,"num_write_kb":2144016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4173,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.009863+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:24:05.009863+0000","last_peered":"2023-10-14T01:24:05.009863+0000","last_clean":"2023-10-14T01:24:05.009863+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:24:05.009863+0000","last_undegraded":"2023-10-14T01:24:05.009863+0000","last_fullsized":"2023-10-14T01:24:05.009863+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'544753","reported_seq":1083280,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:06.725552+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:24:06.725552+0000","last_peered":"2023-10-14T01:24:06.725552+0000","last_clean":"2023-10-14T01:24:06.725552+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:24:06.725552+0000","last_undegraded":"2023-10-14T01:24:06.725552+0000","last_fullsized":"2023-10-14T01:24:06.725552+0000","mapping_epoch":15,"log_start":"96'534727","ondisk_log_start":"96'534727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7470010368,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":540583,"num_read_kb":2131100,"num_write":539470,"num_write_kb":2157880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1513,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:59.898236+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:23:59.898236+0000","last_peered":"2023-10-14T01:23:59.898236+0000","last_clean":"2023-10-14T01:23:59.898236+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:23:59.898236+0000","last_undegraded":"2023-10-14T01:23:59.898236+0000","last_fullsized":"2023-10-14T01:23:59.898236+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'540328","reported_seq":1076531,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:07.916235+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:24:07.916235+0000","last_peered":"2023-10-14T01:24:07.916235+0000","last_clean":"2023-10-14T01:24:07.916235+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:24:07.916235+0000","last_undegraded":"2023-10-14T01:24:07.916235+0000","last_fullsized":"2023-10-14T01:24:07.916235+0000","mapping_epoch":15,"log_start":"96'530234","ondisk_log_start":"96'530234","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":1928,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 301s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7449788435,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":538261,"num_read_kb":2121304,"num_write":536432,"num_write_kb":2145722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'14571","reported_seq":19850,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:05.762881+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:24:05.762881+0000","last_peered":"2023-10-14T01:24:05.762881+0000","last_clean":"2023-10-14T01:24:05.762881+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:24:05.762881+0000","last_undegraded":"2023-10-14T01:24:05.762881+0000","last_fullsized":"2023-10-14T01:24:05.762881+0000","mapping_epoch":12,"log_start":"81'4500","ondisk_log_start":"81'4500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:12.687920+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:23:12.687920+0000","last_peered":"2023-10-14T01:23:12.687920+0000","last_clean":"2023-10-14T01:23:12.687920+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:23:12.687920+0000","last_undegraded":"2023-10-14T01:23:12.687920+0000","last_fullsized":"2023-10-14T01:23:12.687920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29674864659,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2143473,"num_read_kb":8448080,"num_write":2139764,"num_write_kb":8559050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17526587392,"data_stored":20594614310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40262,"ondisk_log_size":40262,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47873,"ondisk_log_size":47873,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739325,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10364256,"kb_used_data":8556824,"kb_used_omap":1425,"kb_used_meta":1805998,"kb_avail":84007584,"statfs":{"total":96636764160,"available":86023766016,"internally_reserved":0,"allocated":8762187776,"data_stored":10295665508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":1849342727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.41999999999999998,"15min":0.36199999999999999},"min":{"1min":0.183,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.35199999999999998,"15min":0.35299999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.7529999999999999,"5min":2.9449999999999998,"15min":3.7519999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.52300000000000002,"15min":0.47799999999999998},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.5960000000000001,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.637,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.36499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739326,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6328524,"kb_used_data":4291332,"kb_used_omap":512,"kb_used_meta":2036671,"kb_avail":88043316,"statfs":{"total":96636764160,"available":90156355584,"internally_reserved":0,"allocated":4394323968,"data_stored":5161381603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":524859,"internal_metadata":2085551557},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,3,4,8,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.55500000000000005,"15min":0.51100000000000001},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.152},"max":{"1min":0.99399999999999999,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.318},{"interface":"front","average":{"1min":0.317,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":0.629,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.19700000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.49099999999999999,"15min":0.44800000000000001},"min":{"1min":0.187,"5min":0.186,"15min":0.186},"max":{"1min":1.034,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.54200000000000004,"15min":0.496},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.1040000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.379}]}]},{"osd":0,"up_from":8,"seq":34359739328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6124544,"kb_used_data":4270200,"kb_used_omap":753,"kb_used_meta":1853582,"kb_avail":88247296,"statfs":{"total":96636764160,"available":90365231104,"internally_reserved":0,"allocated":4372684800,"data_stored":5138193174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1898068725},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27300000000000002,"5min":0.39800000000000002,"15min":0.45700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.48699999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48099999999999998,"15min":0.52700000000000002},"min":{"1min":0.224,"5min":0.191,"15min":0.17399999999999999},"max":{"1min":3.1600000000000001,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":0.20999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.73599999999999999,"15min":0.72799999999999998},"min":{"1min":0.22900000000000001,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":0.85499999999999998,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.246},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.71299999999999997,"15min":0.58199999999999996},"min":{"1min":0.25700000000000001,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.41699999999999998,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4371779584,"data_stored":5137965056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4392701952,"data_stored":5160529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8762105856,"data_stored":10296119315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:24:10.165 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:10.166 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:24:10.735 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:24:10.812 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:23.011 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:24:23.011 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:24:23.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:24:23.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:24:23.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:24:23.012 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:24:23.013 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:24:23.013 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:24:23.013 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:24:23.013 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:24:23.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:24:23.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:24:23.014 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:23.012+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:24:25.120 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:25.120+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:24:25.126 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:25.124+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:24:25.126 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:25.124+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:24:25.126 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:25.124+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:24:25.150 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:25.152+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:24:25.151 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:24:25.152+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:24:29.826 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:24:29.824+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:24:30.814 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:31.263 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:24:31.264 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:24:31.315 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2490,"stamp":"2023-10-14T01:24:30.506192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29708046156,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2174400,"num_read_kb":8500826,"num_write":2148615,"num_write_kb":8593471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88225,"ondisk_log_size":88225,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22985544,"kb_used_data":17182256,"kb_used_omap":2882,"kb_used_meta":5800381,"kb_avail":260129976,"statfs":{"total":289910292480,"available":266373095424,"internally_reserved":0,"allocated":17594630144,"data_stored":20660673885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":5939590157},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,6,4,6,10,21,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16420864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4225,"num_read_kb":16820,"num_write":4051,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002096"},"pg_stats":[{"pgid":"2.7","version":"97'11077","reported_seq":15263,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:27.449820+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:24:27.449820+0000","last_peered":"2023-10-14T01:24:27.449820+0000","last_clean":"2023-10-14T01:24:27.449820+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:24:27.449820+0000","last_undegraded":"2023-10-14T01:24:27.449820+0000","last_fullsized":"2023-10-14T01:24:27.449820+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'7458","reported_seq":11206,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:27.436415+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:24:27.436415+0000","last_peered":"2023-10-14T01:24:27.436415+0000","last_clean":"2023-10-14T01:24:27.436415+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:24:27.436415+0000","last_undegraded":"2023-10-14T01:24:27.436415+0000","last_fullsized":"2023-10-14T01:24:27.436415+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7458,"log_dups_size":0,"ondisk_log_size":7458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":186,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:27.285664+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:23:27.285664+0000","last_peered":"2023-10-14T01:23:27.285664+0000","last_clean":"2023-10-14T01:23:27.285664+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:23:27.285664+0000","last_undegraded":"2023-10-14T01:23:27.285664+0000","last_fullsized":"2023-10-14T01:23:27.285664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"97'14722","reported_seq":20668,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:27.450405+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:24:27.450405+0000","last_peered":"2023-10-14T01:24:27.450405+0000","last_clean":"2023-10-14T01:24:27.450405+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:24:27.450405+0000","last_undegraded":"2023-10-14T01:24:27.450405+0000","last_fullsized":"2023-10-14T01:24:27.450405+0000","mapping_epoch":12,"log_start":"81'4700","ondisk_log_start":"81'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"97'535026","reported_seq":1063100,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.824720+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:24:26.824720+0000","last_peered":"2023-10-14T01:24:26.824720+0000","last_clean":"2023-10-14T01:24:26.824720+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:24:26.824720+0000","last_undegraded":"2023-10-14T01:24:26.824720+0000","last_fullsized":"2023-10-14T01:24:26.824720+0000","mapping_epoch":15,"log_start":"96'524934","ondisk_log_start":"96'524934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7318843392,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":530078,"num_read_kb":2089188,"num_write":529853,"num_write_kb":2119412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"97'10996","reported_seq":15070,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.296850+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:24:26.296850+0000","last_peered":"2023-10-14T01:24:26.296850+0000","last_clean":"2023-10-14T01:24:26.296850+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:24:26.296850+0000","last_undegraded":"2023-10-14T01:24:26.296850+0000","last_fullsized":"2023-10-14T01:24:26.296850+0000","mapping_epoch":12,"log_start":"51'900","ondisk_log_start":"51'900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"97'543260","reported_seq":1079974,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:28.017831+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:24:28.017831+0000","last_peered":"2023-10-14T01:24:28.017831+0000","last_clean":"2023-10-14T01:24:28.017831+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:24:28.017831+0000","last_undegraded":"2023-10-14T01:24:28.017831+0000","last_fullsized":"2023-10-14T01:24:28.017831+0000","mapping_epoch":15,"log_start":"96'533228","ondisk_log_start":"96'533228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7452352512,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":538752,"num_read_kb":2123216,"num_write":537991,"num_write_kb":2151964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4185,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:25.493125+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:24:25.493125+0000","last_peered":"2023-10-14T01:24:25.493125+0000","last_clean":"2023-10-14T01:24:25.493125+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:24:25.493125+0000","last_undegraded":"2023-10-14T01:24:25.493125+0000","last_fullsized":"2023-10-14T01:24:25.493125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'546757","reported_seq":1087445,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:26.797249+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:24:26.797249+0000","last_peered":"2023-10-14T01:24:26.797249+0000","last_clean":"2023-10-14T01:24:26.797249+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:24:26.797249+0000","last_undegraded":"2023-10-14T01:24:26.797249+0000","last_fullsized":"2023-10-14T01:24:26.797249+0000","mapping_epoch":15,"log_start":"96'536727","ondisk_log_start":"96'536727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7478128640,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":542744,"num_read_kb":2139720,"num_write":541474,"num_write_kb":2165896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1517,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:20.373677+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:24:20.373677+0000","last_peered":"2023-10-14T01:24:20.373677+0000","last_clean":"2023-10-14T01:24:20.373677+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:24:20.373677+0000","last_undegraded":"2023-10-14T01:24:20.373677+0000","last_fullsized":"2023-10-14T01:24:20.373677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"97'542476","reported_seq":1080817,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:28.022420+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:24:28.022420+0000","last_peered":"2023-10-14T01:24:28.022420+0000","last_clean":"2023-10-14T01:24:28.022420+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:24:28.022420+0000","last_undegraded":"2023-10-14T01:24:28.022420+0000","last_fullsized":"2023-10-14T01:24:28.022420+0000","mapping_epoch":15,"log_start":"96'532434","ondisk_log_start":"96'532434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2060,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 321s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7458127891,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":540389,"num_read_kb":2129788,"num_write":538487,"num_write_kb":2153942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'14635","reported_seq":19931,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:27.525770+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:24:27.525770+0000","last_peered":"2023-10-14T01:24:27.525770+0000","last_clean":"2023-10-14T01:24:27.525770+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:24:27.525770+0000","last_undegraded":"2023-10-14T01:24:27.525770+0000","last_fullsized":"2023-10-14T01:24:27.525770+0000","mapping_epoch":12,"log_start":"81'4600","ondisk_log_start":"81'4600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":274,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T01:23:12.687920+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:23:12.687920+0000","last_peered":"2023-10-14T01:23:12.687920+0000","last_clean":"2023-10-14T01:23:12.687920+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:23:12.687920+0000","last_undegraded":"2023-10-14T01:23:12.687920+0000","last_fullsized":"2023-10-14T01:23:12.687920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29707452435,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2151963,"num_read_kb":8481912,"num_write":2147805,"num_write_kb":8591214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17592532992,"data_stored":20660559910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47930,"ondisk_log_size":47930,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739329,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10500308,"kb_used_data":8588748,"kb_used_omap":1425,"kb_used_meta":1910126,"kb_avail":83871532,"statfs":{"total":96636764160,"available":85884448768,"internally_reserved":0,"allocated":8794877952,"data_stored":10328355684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":1955969799},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.41999999999999998,"15min":0.36199999999999999},"min":{"1min":0.183,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.35199999999999998,"15min":0.35299999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.7529999999999999,"5min":2.9449999999999998,"15min":3.7519999999999998},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.52300000000000002,"15min":0.47799999999999998},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.5960000000000001,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.637,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739330,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6308620,"kb_used_data":4307332,"kb_used_omap":704,"kb_used_meta":2000575,"kb_avail":88063220,"statfs":{"total":96636764160,"available":90176737280,"internally_reserved":0,"allocated":4410707968,"data_stored":5177765603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":2048589329},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,1,7,19,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.55500000000000005,"15min":0.51100000000000001},"min":{"1min":0.16900000000000001,"5min":0.158,"15min":0.152},"max":{"1min":0.99399999999999999,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.432},{"interface":"front","average":{"1min":0.317,"5min":0.52100000000000002,"15min":0.53600000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":0.629,"5min":5.3360000000000003,"15min":8.8089999999999993},"last":0.215}]},{"osd":2,"last update":"Sat Oct 14 01:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.49099999999999999,"15min":0.44800000000000001},"min":{"1min":0.187,"5min":0.186,"15min":0.186},"max":{"1min":1.034,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.373},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.54200000000000004,"15min":0.496},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.1040000000000001,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.40100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739332,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6176616,"kb_used_data":4286176,"kb_used_omap":753,"kb_used_meta":1889678,"kb_avail":88195224,"statfs":{"total":96636764160,"available":90311909376,"internally_reserved":0,"allocated":4389044224,"data_stored":5154552598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1935031029},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42399999999999999,"15min":0.45800000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.7549999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.215},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.46600000000000003,"15min":0.52500000000000002},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.56000000000000005,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":1.325}]},{"osd":2,"last update":"Sat Oct 14 01:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.754,"15min":0.67500000000000004},"min":{"1min":0.20399999999999999,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":1.7649999999999999,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.70799999999999996,"15min":0.55300000000000005},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.44,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4388163584,"data_stored":5154349056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4409257984,"data_stored":5177085971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8795111424,"data_stored":10329124883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:24:31.317 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:31.317 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:24:51.319 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:24:51.797 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:24:51.798 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:24:51.821 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2501,"stamp":"2023-10-14T01:24:50.509405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29741637452,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2183000,"num_read_kb":8535094,"num_write":2156933,"num_write_kb":8626743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88162,"ondisk_log_size":88162,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23213752,"kb_used_data":17248736,"kb_used_omap":2882,"kb_used_meta":5962109,"kb_avail":259901768,"statfs":{"total":289910292480,"available":266139410432,"internally_reserved":0,"allocated":17662705664,"data_stored":20728735557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":6105199629},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,13,10,7,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16723968,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4358,"num_read_kb":17376,"num_write":4134,"num_write_kb":16536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001919"},"pg_stats":[{"pgid":"2.7","version":"98'11104","reported_seq":15303,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:45.969751+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:24:45.969751+0000","last_peered":"2023-10-14T01:24:45.969751+0000","last_clean":"2023-10-14T01:24:45.969751+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:24:45.969751+0000","last_undegraded":"2023-10-14T01:24:45.969751+0000","last_fullsized":"2023-10-14T01:24:45.969751+0000","mapping_epoch":12,"log_start":"53'1100","ondisk_log_start":"53'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'7476","reported_seq":11233,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:45.969618+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:24:45.969618+0000","last_peered":"2023-10-14T01:24:45.969618+0000","last_clean":"2023-10-14T01:24:45.969618+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:24:45.969618+0000","last_undegraded":"2023-10-14T01:24:45.969618+0000","last_fullsized":"2023-10-14T01:24:45.969618+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":187,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:28.036254+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:24:28.036254+0000","last_peered":"2023-10-14T01:24:28.036254+0000","last_clean":"2023-10-14T01:24:28.036254+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:24:28.036254+0000","last_undegraded":"2023-10-14T01:24:28.036254+0000","last_fullsized":"2023-10-14T01:24:28.036254+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'14758","reported_seq":20721,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:45.969937+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:24:45.969937+0000","last_peered":"2023-10-14T01:24:45.969937+0000","last_clean":"2023-10-14T01:24:45.969937+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:24:45.969937+0000","last_undegraded":"2023-10-14T01:24:45.969937+0000","last_fullsized":"2023-10-14T01:24:45.969937+0000","mapping_epoch":12,"log_start":"81'4700","ondisk_log_start":"81'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'537072","reported_seq":1067332,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:46.959559+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:24:46.959559+0000","last_peered":"2023-10-14T01:24:46.959559+0000","last_clean":"2023-10-14T01:24:46.959559+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:24:46.959559+0000","last_undegraded":"2023-10-14T01:24:46.959559+0000","last_fullsized":"2023-10-14T01:24:46.959559+0000","mapping_epoch":15,"log_start":"96'527034","ondisk_log_start":"96'527034","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7327064064,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":532263,"num_read_kb":2097896,"num_write":531899,"num_write_kb":2127596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'11032","reported_seq":15119,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:45.970318+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:24:45.970318+0000","last_peered":"2023-10-14T01:24:45.970318+0000","last_clean":"2023-10-14T01:24:45.970318+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:24:45.970318+0000","last_undegraded":"2023-10-14T01:24:45.970318+0000","last_fullsized":"2023-10-14T01:24:45.970318+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'545392","reported_seq":1084172,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:48.118459+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:24:48.118459+0000","last_peered":"2023-10-14T01:24:48.118459+0000","last_clean":"2023-10-14T01:24:48.118459+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:24:48.118459+0000","last_undegraded":"2023-10-14T01:24:48.118459+0000","last_fullsized":"2023-10-14T01:24:48.118459+0000","mapping_epoch":15,"log_start":"96'535328","ondisk_log_start":"96'535328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7461003264,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":540816,"num_read_kb":2131432,"num_write":540123,"num_write_kb":2160492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4198,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:45.971230+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:24:45.971230+0000","last_peered":"2023-10-14T01:24:45.971230+0000","last_clean":"2023-10-14T01:24:45.971230+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:24:45.971230+0000","last_undegraded":"2023-10-14T01:24:45.971230+0000","last_fullsized":"2023-10-14T01:24:45.971230+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'548796","reported_seq":1091640,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:46.969346+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:24:46.969346+0000","last_peered":"2023-10-14T01:24:46.969346+0000","last_clean":"2023-10-14T01:24:46.969346+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:24:46.969346+0000","last_undegraded":"2023-10-14T01:24:46.969346+0000","last_fullsized":"2023-10-14T01:24:46.969346+0000","mapping_epoch":15,"log_start":"96'538727","ondisk_log_start":"96'538727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7486386176,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":544901,"num_read_kb":2148300,"num_write":543513,"num_write_kb":2174052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1522,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:40.853261+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:24:40.853261+0000","last_peered":"2023-10-14T01:24:40.853261+0000","last_clean":"2023-10-14T01:24:40.853261+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:24:40.853261+0000","last_undegraded":"2023-10-14T01:24:40.853261+0000","last_fullsized":"2023-10-14T01:24:40.853261+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'544643","reported_seq":1085195,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:24:48.119744+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:24:48.119744+0000","last_peered":"2023-10-14T01:24:48.119744+0000","last_clean":"2023-10-14T01:24:48.119744+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:24:48.119744+0000","last_undegraded":"2023-10-14T01:24:48.119744+0000","last_fullsized":"2023-10-14T01:24:48.119744+0000","mapping_epoch":15,"log_start":"96'534634","ondisk_log_start":"96'534634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2156,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 342s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7466590227,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":542583,"num_read_kb":2138552,"num_write":540588,"num_write_kb":2162346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'14671","reported_seq":19984,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:45.969989+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:24:45.969989+0000","last_peered":"2023-10-14T01:24:45.969989+0000","last_clean":"2023-10-14T01:24:45.969989+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:24:45.969989+0000","last_undegraded":"2023-10-14T01:24:45.969989+0000","last_fullsized":"2023-10-14T01:24:45.969989+0000","mapping_epoch":12,"log_start":"81'4600","ondisk_log_start":"81'4600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":275,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:27.285491+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:23:27.285491+0000","last_peered":"2023-10-14T01:23:27.285491+0000","last_clean":"2023-10-14T01:23:27.285491+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:23:27.285491+0000","last_undegraded":"2023-10-14T01:23:27.285491+0000","last_fullsized":"2023-10-14T01:23:27.285491+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29741043731,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2160563,"num_read_kb":8516180,"num_write":2156123,"num_write_kb":8624486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17660477440,"data_stored":20728504358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40180,"ondisk_log_size":40180,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47883,"ondisk_log_size":47883,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739333,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10641404,"kb_used_data":8621620,"kb_used_omap":1425,"kb_used_meta":2018350,"kb_avail":83730436,"statfs":{"total":96636764160,"available":85739966464,"internally_reserved":0,"allocated":8828538880,"data_stored":10362011996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":2066791175},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.41999999999999998,"15min":0.36199999999999999},"min":{"1min":0.183,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":0.60199999999999998,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.35199999999999998,"15min":0.35299999999999998},"min":{"1min":0.218,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.7529999999999999,"5min":2.9449999999999998,"15min":3.7519999999999998},"last":0.245}]},{"osd":1,"last update":"Sat Oct 14 01:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.52300000000000002,"15min":0.47799999999999998},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.5960000000000001,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.57899999999999996,"15min":0.52600000000000002},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.637,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.29299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739334,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6361464,"kb_used_data":4324080,"kb_used_omap":704,"kb_used_meta":2036671,"kb_avail":88010376,"statfs":{"total":96636764160,"available":90122625024,"internally_reserved":0,"allocated":4427857920,"data_stored":5194910939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":2085551633},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,6,3,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.53700000000000003,"15min":0.504},"min":{"1min":0.189,"5min":0.158,"15min":0.152},"max":{"1min":0.76400000000000001,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.496,"15min":0.50800000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":5.1310000000000002,"5min":5.1310000000000002,"15min":5.8879999999999999},"last":0.223}]},{"osd":2,"last update":"Sat Oct 14 01:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.49299999999999999,"15min":0.44500000000000001},"min":{"1min":0.23599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.48999999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.54100000000000004,"15min":0.49299999999999999},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53400000000000003,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.40999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739336,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6210884,"kb_used_data":4303036,"kb_used_omap":753,"kb_used_meta":1907086,"kb_avail":88160956,"statfs":{"total":96636764160,"available":90276818944,"internally_reserved":0,"allocated":4406308864,"data_stored":5171812622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1952856821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,10,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42399999999999999,"15min":0.45800000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.7549999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.46600000000000003,"15min":0.52500000000000002},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.56000000000000005,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":0.254}]},{"osd":2,"last update":"Sat Oct 14 01:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.754,"15min":0.67500000000000004},"min":{"1min":0.20399999999999999,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":1.7649999999999999,"5min":31.103000000000002,"15min":31.103000000000002},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.70799999999999996,"15min":0.55300000000000005},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.44,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.34799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4404899840,"data_stored":5171085312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4426588160,"data_stored":5194416147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8828989440,"data_stored":10363002899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:24:51.824 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:24:51.824 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:02.404 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.399+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:25:02.415 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.411+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:25:02.415 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.411+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:25:02.435 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.431+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:25:02.435 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.431+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:25:02.446 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.443+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:25:02.446 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:02.443+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:25:05.007 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:05.007+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:25:05.008 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:05.007+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:25:05.009 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:05.007+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:25:05.010 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:25:05.007+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:25:11.826 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:12.288 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:25:12.289 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:25:12.324 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2511,"stamp":"2023-10-14T01:25:10.512843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29773815628,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2191241,"num_read_kb":8567994,"num_write":2164869,"num_write_kb":8658487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88214,"ondisk_log_size":88214,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22158028,"kb_used_data":17312308,"kb_used_omap":2882,"kb_used_meta":4842813,"kb_avail":260957492,"statfs":{"total":289910292480,"available":267220471808,"internally_reserved":0,"allocated":17727803392,"data_stored":20793833285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":4959040525},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,3,5,8,12,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15593472,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3885,"num_read_kb":15496,"num_write":3875,"num_write_kb":15500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002076"},"pg_stats":[{"pgid":"2.7","version":"98'11156","reported_seq":15367,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.802261+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:25:07.802261+0000","last_peered":"2023-10-14T01:25:07.802261+0000","last_clean":"2023-10-14T01:25:07.802261+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:25:07.802261+0000","last_undegraded":"2023-10-14T01:25:07.802261+0000","last_fullsized":"2023-10-14T01:25:07.802261+0000","mapping_epoch":12,"log_start":"53'1100","ondisk_log_start":"53'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'7511","reported_seq":11276,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.774637+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:25:07.774637+0000","last_peered":"2023-10-14T01:25:07.774637+0000","last_clean":"2023-10-14T01:25:07.774637+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:25:07.774637+0000","last_undegraded":"2023-10-14T01:25:07.774637+0000","last_fullsized":"2023-10-14T01:25:07.774637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":187,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:28.036254+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:24:28.036254+0000","last_peered":"2023-10-14T01:24:28.036254+0000","last_clean":"2023-10-14T01:24:28.036254+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:24:28.036254+0000","last_undegraded":"2023-10-14T01:24:28.036254+0000","last_fullsized":"2023-10-14T01:24:28.036254+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'14827","reported_seq":20806,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.802674+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:25:07.802674+0000","last_peered":"2023-10-14T01:25:07.802674+0000","last_clean":"2023-10-14T01:25:07.802674+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:25:07.802674+0000","last_undegraded":"2023-10-14T01:25:07.802674+0000","last_fullsized":"2023-10-14T01:25:07.802674+0000","mapping_epoch":12,"log_start":"81'4800","ondisk_log_start":"81'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'539006","reported_seq":1071280,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.072264+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:25:07.072264+0000","last_peered":"2023-10-14T01:25:07.072264+0000","last_clean":"2023-10-14T01:25:07.072264+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:25:07.072264+0000","last_undegraded":"2023-10-14T01:25:07.072264+0000","last_fullsized":"2023-10-14T01:25:07.072264+0000","mapping_epoch":15,"log_start":"96'528934","ondisk_log_start":"96'528934","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7334891520,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":534283,"num_read_kb":2105956,"num_write":533833,"num_write_kb":2135332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'11065","reported_seq":15164,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:06.449631+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:25:06.449631+0000","last_peered":"2023-10-14T01:25:06.449631+0000","last_clean":"2023-10-14T01:25:06.449631+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:25:06.449631+0000","last_undegraded":"2023-10-14T01:25:06.449631+0000","last_fullsized":"2023-10-14T01:25:06.449631+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'547347","reported_seq":1088209,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:08.167787+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:25:08.167787+0000","last_peered":"2023-10-14T01:25:08.167787+0000","last_clean":"2023-10-14T01:25:08.167787+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:25:08.167787+0000","last_undegraded":"2023-10-14T01:25:08.167787+0000","last_fullsized":"2023-10-14T01:25:08.167787+0000","mapping_epoch":15,"log_start":"97'537328","ondisk_log_start":"97'537328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7468961792,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":542900,"num_read_kb":2139760,"num_write":542078,"num_write_kb":2168312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4207,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:01.337547+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:25:01.337547+0000","last_peered":"2023-10-14T01:25:01.337547+0000","last_clean":"2023-10-14T01:25:01.337547+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:25:01.337547+0000","last_undegraded":"2023-10-14T01:25:01.337547+0000","last_fullsized":"2023-10-14T01:25:01.337547+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'550866","reported_seq":1095761,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.064775+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:25:07.064775+0000","last_peered":"2023-10-14T01:25:07.064775+0000","last_clean":"2023-10-14T01:25:07.064775+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:25:07.064775+0000","last_undegraded":"2023-10-14T01:25:07.064775+0000","last_fullsized":"2023-10-14T01:25:07.064775+0000","mapping_epoch":15,"log_start":"97'540827","ondisk_log_start":"97'540827","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7494725632,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":546954,"num_read_kb":2156496,"num_write":545583,"num_write_kb":2182332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1525,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:56.213871+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:24:56.213871+0000","last_peered":"2023-10-14T01:24:56.213871+0000","last_clean":"2023-10-14T01:24:56.213871+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:24:56.213871+0000","last_undegraded":"2023-10-14T01:24:56.213871+0000","last_fullsized":"2023-10-14T01:24:56.213871+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'546678","reported_seq":1089326,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:08.193324+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:25:08.193324+0000","last_peered":"2023-10-14T01:25:08.193324+0000","last_clean":"2023-10-14T01:25:08.193324+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:25:08.193324+0000","last_undegraded":"2023-10-14T01:25:08.193324+0000","last_fullsized":"2023-10-14T01:25:08.193324+0000","mapping_epoch":15,"log_start":"97'536634","ondisk_log_start":"97'536634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2243,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 362s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7474642963,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":544667,"num_read_kb":2146868,"num_write":542565,"num_write_kb":2170254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'14740","reported_seq":20069,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:07.827480+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:25:07.827480+0000","last_peered":"2023-10-14T01:25:07.827480+0000","last_clean":"2023-10-14T01:25:07.827480+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:25:07.827480+0000","last_undegraded":"2023-10-14T01:25:07.827480+0000","last_fullsized":"2023-10-14T01:25:07.827480+0000","mapping_epoch":12,"log_start":"81'4700","ondisk_log_start":"81'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":275,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:23:27.285491+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:23:27.285491+0000","last_peered":"2023-10-14T01:23:27.285491+0000","last_clean":"2023-10-14T01:23:27.285491+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:23:27.285491+0000","last_undegraded":"2023-10-14T01:23:27.285491+0000","last_fullsized":"2023-10-14T01:23:27.285491+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29773221907,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2168804,"num_read_kb":8549080,"num_write":2164059,"num_write_kb":8656230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17725538304,"data_stored":20793565222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40174,"ondisk_log_size":40174,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":47941,"ondisk_log_size":47941,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739337,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10690844,"kb_used_data":8653524,"kb_used_omap":1425,"kb_used_meta":2035886,"kb_avail":83680996,"statfs":{"total":96636764160,"available":85689339904,"internally_reserved":0,"allocated":8861208576,"data_stored":10394681692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":2084748039},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.27700000000000002,"5min":0.39700000000000002,"15min":0.35899999999999999},"min":{"1min":0.191,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.46500000000000002,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.33200000000000002,"15min":0.34799999999999998},"min":{"1min":0.216,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":0.71299999999999997,"5min":1.7529999999999999,"15min":3.7519999999999998},"last":0.28399999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.224,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":4.7789999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.317},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.53800000000000003},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.8120000000000003,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.38200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739338,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5204328,"kb_used_data":4339872,"kb_used_omap":704,"kb_used_meta":863743,"kb_avail":89167512,"statfs":{"total":96636764160,"available":91307532288,"internally_reserved":0,"allocated":4444028928,"data_stored":5211081947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":884473361},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,0,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.53700000000000003,"15min":0.504},"min":{"1min":0.189,"5min":0.158,"15min":0.152},"max":{"1min":0.76400000000000001,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.17799999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.496,"15min":0.50800000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":5.1310000000000002,"5min":5.1310000000000002,"15min":5.8879999999999999},"last":0.255}]},{"osd":2,"last update":"Sat Oct 14 01:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.49299999999999999,"15min":0.44500000000000001},"min":{"1min":0.23599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.48999999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.54100000000000004,"15min":0.49299999999999999},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53400000000000003,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.32600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739340,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6262856,"kb_used_data":4318912,"kb_used_omap":753,"kb_used_meta":1943182,"kb_avail":88108984,"statfs":{"total":96636764160,"available":90223599616,"internally_reserved":0,"allocated":4422565888,"data_stored":5188069646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1989819125},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,10,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42399999999999999,"15min":0.45800000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.7549999999999999,"5min":4.8220000000000001,"15min":6.1130000000000004},"last":0.219},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.46600000000000003,"15min":0.52500000000000002},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.56000000000000005,"5min":4.6260000000000003,"15min":6.8369999999999997},"last":0.23599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.754,"15min":0.67500000000000004},"min":{"1min":0.20399999999999999,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":1.7649999999999999,"5min":31.103000000000002,"15min":31.103000000000002},"last":1.1870000000000001},{"interface":"front","average":{"1min":0.29699999999999999,"5min":0.70799999999999996,"15min":0.55300000000000005},"min":{"1min":0.219,"5min":0.20999999999999999,"15min":0.20399999999999999},"max":{"1min":0.44,"5min":30.234000000000002,"15min":30.234000000000002},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4421296128,"data_stored":5187481600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4442705920,"data_stored":5210533907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8861536256,"data_stored":10395549715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:25:12.325 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:12.326 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:32.327 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:33.044 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:25:33.045 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:25:33.078 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2522,"stamp":"2023-10-14T01:25:32.516367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29810212684,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2200477,"num_read_kb":8604806,"num_write":2173861,"num_write_kb":8694455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88418,"ondisk_log_size":88418,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":20762400,"kb_used_data":17388360,"kb_used_omap":2882,"kb_used_meta":3371133,"kb_avail":262353120,"statfs":{"total":289910292480,"available":268649594880,"internally_reserved":0,"allocated":17805680640,"data_stored":20871710533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":3452040205},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5,9,3,11,17],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20004864,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5120,"num_read_kb":20388,"num_write":4941,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001908"},"pg_stats":[{"pgid":"2.7","version":"98'11201","reported_seq":15425,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:26.934357+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:25:26.934357+0000","last_peered":"2023-10-14T01:25:26.934357+0000","last_clean":"2023-10-14T01:25:26.934357+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:25:26.934357+0000","last_undegraded":"2023-10-14T01:25:26.934357+0000","last_fullsized":"2023-10-14T01:25:26.934357+0000","mapping_epoch":12,"log_start":"53'1200","ondisk_log_start":"53'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'7541","reported_seq":11315,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:26.933385+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:25:26.933385+0000","last_peered":"2023-10-14T01:25:26.933385+0000","last_clean":"2023-10-14T01:25:26.933385+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:25:26.933385+0000","last_undegraded":"2023-10-14T01:25:26.933385+0000","last_fullsized":"2023-10-14T01:25:26.933385+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7541,"log_dups_size":0,"ondisk_log_size":7541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":187,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-14T01:24:28.036254+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:24:28.036254+0000","last_peered":"2023-10-14T01:24:28.036254+0000","last_clean":"2023-10-14T01:24:28.036254+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:24:28.036254+0000","last_undegraded":"2023-10-14T01:24:28.036254+0000","last_fullsized":"2023-10-14T01:24:28.036254+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'14887","reported_seq":20882,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:26.933922+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:25:26.933922+0000","last_peered":"2023-10-14T01:25:26.933922+0000","last_clean":"2023-10-14T01:25:26.933922+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:25:26.933922+0000","last_undegraded":"2023-10-14T01:25:26.933922+0000","last_fullsized":"2023-10-14T01:25:26.933922+0000","mapping_epoch":12,"log_start":"81'4800","ondisk_log_start":"81'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"98'541473","reported_seq":1076242,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:32.166495+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:25:32.166495+0000","last_peered":"2023-10-14T01:25:32.166495+0000","last_clean":"2023-10-14T01:25:32.166495+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:25:32.166495+0000","last_undegraded":"2023-10-14T01:25:32.166495+0000","last_fullsized":"2023-10-14T01:25:32.166495+0000","mapping_epoch":15,"log_start":"97'531434","ondisk_log_start":"97'531434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7344857088,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":536771,"num_read_kb":2115876,"num_write":536300,"num_write_kb":2145200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"98'11095","reported_seq":15209,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:32.049638+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:25:32.049638+0000","last_peered":"2023-10-14T01:25:32.049638+0000","last_clean":"2023-10-14T01:25:32.049638+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:25:32.049638+0000","last_undegraded":"2023-10-14T01:25:32.049638+0000","last_fullsized":"2023-10-14T01:25:32.049638+0000","mapping_epoch":12,"log_start":"51'1000","ondisk_log_start":"51'1000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"98'549328","reported_seq":1092257,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:28.278356+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:25:28.278356+0000","last_peered":"2023-10-14T01:25:28.278356+0000","last_clean":"2023-10-14T01:25:28.278356+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:25:28.278356+0000","last_undegraded":"2023-10-14T01:25:28.278356+0000","last_fullsized":"2023-10-14T01:25:28.278356+0000","mapping_epoch":15,"log_start":"97'539228","ondisk_log_start":"97'539228","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7476953088,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":544969,"num_read_kb":2148004,"num_write":544059,"num_write_kb":2176236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4222,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:27.075591+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:25:27.075591+0000","last_peered":"2023-10-14T01:25:27.075591+0000","last_clean":"2023-10-14T01:25:27.075591+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:25:27.075591+0000","last_undegraded":"2023-10-14T01:25:27.075591+0000","last_fullsized":"2023-10-14T01:25:27.075591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"98'553361","reported_seq":1100819,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:32.146803+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:25:32.146803+0000","last_peered":"2023-10-14T01:25:32.146803+0000","last_clean":"2023-10-14T01:25:32.146803+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:25:32.146803+0000","last_undegraded":"2023-10-14T01:25:32.146803+0000","last_fullsized":"2023-10-14T01:25:32.146803+0000","mapping_epoch":15,"log_start":"97'543327","ondisk_log_start":"97'543327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7504793600,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":549513,"num_read_kb":2166700,"num_write":548078,"num_write_kb":2192312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1531,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:21.837854+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:25:21.837854+0000","last_peered":"2023-10-14T01:25:21.837854+0000","last_clean":"2023-10-14T01:25:21.837854+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:25:21.837854+0000","last_undegraded":"2023-10-14T01:25:21.837854+0000","last_fullsized":"2023-10-14T01:25:21.837854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'548814","reported_seq":1093589,"reported_epoch":98,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:28.284403+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:25:28.284403+0000","last_peered":"2023-10-14T01:25:28.284403+0000","last_clean":"2023-10-14T01:25:28.284403+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:25:28.284403+0000","last_undegraded":"2023-10-14T01:25:28.284403+0000","last_fullsized":"2023-10-14T01:25:28.284403+0000","mapping_epoch":15,"log_start":"97'538734","ondisk_log_start":"97'538734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2366,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 382s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7483015187,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":546787,"num_read_kb":2155312,"num_write":544614,"num_write_kb":2178450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'14800","reported_seq":20145,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:26.934454+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:25:26.934454+0000","last_peered":"2023-10-14T01:25:26.934454+0000","last_clean":"2023-10-14T01:25:26.934454+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:25:26.934454+0000","last_undegraded":"2023-10-14T01:25:26.934454+0000","last_fullsized":"2023-10-14T01:25:26.934454+0000","mapping_epoch":12,"log_start":"81'4700","ondisk_log_start":"81'4700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":276,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:37.332604+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:24:37.332604+0000","last_peered":"2023-10-14T01:24:37.332604+0000","last_clean":"2023-10-14T01:24:37.332604+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:24:37.332604+0000","last_undegraded":"2023-10-14T01:24:37.332604+0000","last_fullsized":"2023-10-14T01:24:37.332604+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29809618963,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2178040,"num_read_kb":8585892,"num_write":2173051,"num_write_kb":8692198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17803288576,"data_stored":20871315494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40253,"ondisk_log_size":40253,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48066,"ondisk_log_size":48066,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739342,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10319984,"kb_used_data":8693736,"kb_used_omap":1425,"kb_used_meta":1624814,"kb_avail":84051856,"statfs":{"total":96636764160,"available":86069100544,"internally_reserved":0,"allocated":8902385664,"data_stored":10435858780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":1663810311},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.27700000000000002,"5min":0.39700000000000002,"15min":0.35899999999999999},"min":{"1min":0.191,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.46500000000000002,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.33200000000000002,"15min":0.34799999999999998},"min":{"1min":0.216,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":0.71299999999999997,"5min":1.7529999999999999,"15min":3.7519999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.48899999999999999},"min":{"1min":0.224,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":4.7789999999999999,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.59499999999999997,"15min":0.53800000000000003},"min":{"1min":0.28299999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.8120000000000003,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.27400000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739342,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5256476,"kb_used_data":4355924,"kb_used_omap":704,"kb_used_meta":899839,"kb_avail":89115364,"statfs":{"total":96636764160,"available":91254132736,"internally_reserved":0,"allocated":4460466176,"data_stored":5227519195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":921435665},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,6,2,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.53700000000000003,"15min":0.504},"min":{"1min":0.189,"5min":0.158,"15min":0.152},"max":{"1min":0.76400000000000001,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.496,"15min":0.50800000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":5.1310000000000002,"5min":5.1310000000000002,"15min":5.8879999999999999},"last":0.224}]},{"osd":2,"last update":"Sat Oct 14 01:24:29 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.49299999999999999,"15min":0.44500000000000001},"min":{"1min":0.23599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.48999999999999999,"5min":6.0709999999999997,"15min":6.0709999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.54100000000000004,"15min":0.49299999999999999},"min":{"1min":0.26000000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53400000000000003,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.26800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739345,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5185940,"kb_used_data":4338700,"kb_used_omap":753,"kb_used_meta":846478,"kb_avail":89185900,"statfs":{"total":96636764160,"available":91326361600,"internally_reserved":0,"allocated":4442828800,"data_stored":5208332558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":866794229},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,1,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.44900000000000001},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.3279999999999998,"5min":2.7549999999999999,"15min":6.1130000000000004},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.49399999999999999,"15min":0.54200000000000004},"min":{"1min":0.19700000000000001,"5min":0.191,"15min":0.191},"max":{"1min":6.2320000000000002,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.42299999999999999,"15min":0.65100000000000002},"min":{"1min":0.22700000000000001,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":1.1870000000000001,"5min":4.6120000000000001,"15min":31.103000000000002},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.38,"15min":0.54000000000000004},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.52400000000000002,"5min":4.5419999999999998,"15min":30.234000000000002},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4441612288,"data_stored":5207797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4459200512,"data_stored":5227028499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8902475776,"data_stored":10436489235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:25:33.080 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:33.081 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:25:53.082 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:25:53.647 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:25:53.648 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:25:53.722 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2533,"stamp":"2023-10-14T01:25:52.519703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29843554124,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2208740,"num_read_kb":8637774,"num_write":2182070,"num_write_kb":8727291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88336,"ondisk_log_size":88336,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21015260,"kb_used_data":17453952,"kb_used_omap":2882,"kb_used_meta":3558397,"kb_avail":262100260,"statfs":{"total":289910292480,"available":268390666240,"internally_reserved":0,"allocated":17872846848,"data_stored":20938862764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":3643798541},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,6,7,8,12,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17186816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4217,"num_read_kb":16828,"num_write":4206,"num_write_kb":16824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001695"},"pg_stats":[{"pgid":"2.7","version":"98'11270","reported_seq":15507,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.410251+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:25:47.410251+0000","last_peered":"2023-10-14T01:25:47.410251+0000","last_clean":"2023-10-14T01:25:47.410251+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:25:47.410251+0000","last_undegraded":"2023-10-14T01:25:47.410251+0000","last_fullsized":"2023-10-14T01:25:47.410251+0000","mapping_epoch":12,"log_start":"53'1200","ondisk_log_start":"53'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"98'7587","reported_seq":11370,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.409575+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:25:47.409575+0000","last_peered":"2023-10-14T01:25:47.409575+0000","last_clean":"2023-10-14T01:25:47.409575+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:25:47.409575+0000","last_undegraded":"2023-10-14T01:25:47.409575+0000","last_fullsized":"2023-10-14T01:25:47.409575+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7587,"log_dups_size":0,"ondisk_log_size":7587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":188,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:37.324639+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:24:37.324639+0000","last_peered":"2023-10-14T01:24:37.324639+0000","last_clean":"2023-10-14T01:24:37.324639+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:24:37.324639+0000","last_undegraded":"2023-10-14T01:24:37.324639+0000","last_fullsized":"2023-10-14T01:24:37.324639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"98'14979","reported_seq":20991,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.410299+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:25:47.410299+0000","last_peered":"2023-10-14T01:25:47.410299+0000","last_clean":"2023-10-14T01:25:47.410299+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:25:47.410299+0000","last_undegraded":"2023-10-14T01:25:47.410299+0000","last_fullsized":"2023-10-14T01:25:47.410299+0000","mapping_epoch":12,"log_start":"81'4900","ondisk_log_start":"81'4900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'543483","reported_seq":1080363,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:52.278229+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:25:52.278229+0000","last_peered":"2023-10-14T01:25:52.278229+0000","last_clean":"2023-10-14T01:25:52.278229+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:25:52.278229+0000","last_undegraded":"2023-10-14T01:25:52.278229+0000","last_fullsized":"2023-10-14T01:25:52.278229+0000","mapping_epoch":15,"log_start":"97'533434","ondisk_log_start":"97'533434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7353061376,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":538882,"num_read_kb":2124300,"num_write":538310,"num_write_kb":2153240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11120","reported_seq":15244,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:52.151091+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:25:52.151091+0000","last_peered":"2023-10-14T01:25:52.151091+0000","last_clean":"2023-10-14T01:25:52.151091+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:25:52.151091+0000","last_undegraded":"2023-10-14T01:25:52.151091+0000","last_fullsized":"2023-10-14T01:25:52.151091+0000","mapping_epoch":12,"log_start":"53'1100","ondisk_log_start":"53'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'551340","reported_seq":1096283,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:48.391286+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:25:48.391286+0000","last_peered":"2023-10-14T01:25:48.391286+0000","last_clean":"2023-10-14T01:25:48.391286+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:25:48.391286+0000","last_undegraded":"2023-10-14T01:25:48.391286+0000","last_fullsized":"2023-10-14T01:25:48.391286+0000","mapping_epoch":15,"log_start":"97'541328","ondisk_log_start":"97'541328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7485100032,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":546980,"num_read_kb":2156020,"num_write":546071,"num_write_kb":2184284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4235,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.410382+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:25:47.410382+0000","last_peered":"2023-10-14T01:25:47.410382+0000","last_clean":"2023-10-14T01:25:47.410382+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:25:47.410382+0000","last_undegraded":"2023-10-14T01:25:47.410382+0000","last_fullsized":"2023-10-14T01:25:47.410382+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'555428","reported_seq":1104988,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:52.279996+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:25:52.279996+0000","last_peered":"2023-10-14T01:25:52.279996+0000","last_clean":"2023-10-14T01:25:52.279996+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:25:52.279996+0000","last_undegraded":"2023-10-14T01:25:52.279996+0000","last_fullsized":"2023-10-14T01:25:52.279996+0000","mapping_epoch":15,"log_start":"97'545427","ondisk_log_start":"97'545427","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7513231360,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":551614,"num_read_kb":2175084,"num_write":550145,"num_write_kb":2200580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1536,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.344390+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:25:47.344390+0000","last_peered":"2023-10-14T01:25:47.344390+0000","last_clean":"2023-10-14T01:25:47.344390+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:25:47.344390+0000","last_undegraded":"2023-10-14T01:25:47.344390+0000","last_fullsized":"2023-10-14T01:25:47.344390+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'551020","reported_seq":1097854,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:25:48.386256+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:25:48.386256+0000","last_peered":"2023-10-14T01:25:48.386256+0000","last_clean":"2023-10-14T01:25:48.386256+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:25:48.386256+0000","last_undegraded":"2023-10-14T01:25:48.386256+0000","last_fullsized":"2023-10-14T01:25:48.386256+0000","mapping_epoch":15,"log_start":"97'540935","ondisk_log_start":"97'540935","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2492,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 402s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7491567635,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":548827,"num_read_kb":2163456,"num_write":546734,"num_write_kb":2186930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"98'14892","reported_seq":20254,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.410584+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:25:47.410584+0000","last_peered":"2023-10-14T01:25:47.410584+0000","last_clean":"2023-10-14T01:25:47.410584+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:25:47.410584+0000","last_undegraded":"2023-10-14T01:25:47.410584+0000","last_fullsized":"2023-10-14T01:25:47.410584+0000","mapping_epoch":12,"log_start":"81'4800","ondisk_log_start":"81'4800","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":277,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:13.231536+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:13.231536+0000","last_peered":"2023-10-14T01:25:13.231536+0000","last_clean":"2023-10-14T01:25:13.231536+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:13.231536+0000","last_undegraded":"2023-10-14T01:25:13.231536+0000","last_fullsized":"2023-10-14T01:25:13.231536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29842960403,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2186303,"num_read_kb":8618860,"num_write":2181260,"num_write_kb":8725034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17870708736,"data_stored":20938735654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40147,"ondisk_log_size":40147,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48090,"ondisk_log_size":48090,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739346,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10471528,"kb_used_data":8727072,"kb_used_omap":1425,"kb_used_meta":1743022,"kb_avail":83900312,"statfs":{"total":96636764160,"available":85913919488,"internally_reserved":0,"allocated":8936521728,"data_stored":10469990185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":1784855303},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:48 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.40200000000000002,"15min":0.34899999999999998},"min":{"1min":0.21299999999999999,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.54300000000000004,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.35699999999999998,"15min":0.34799999999999998},"min":{"1min":0.23999999999999999,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.1679999999999999,"5min":1.7529999999999999,"15min":3.7519999999999998},"last":0.23999999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:25:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.51800000000000002,"15min":0.47099999999999997},"min":{"1min":0.23400000000000001,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.782,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.57499999999999996,"15min":0.52000000000000002},"min":{"1min":0.27400000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.8200000000000001,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.32600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739346,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5301396,"kb_used_data":4372040,"kb_used_omap":704,"kb_used_meta":928639,"kb_avail":89070444,"statfs":{"total":96636764160,"available":91208134656,"internally_reserved":0,"allocated":4476968960,"data_stored":5244017320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":950926865},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,4,2,11,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.50900000000000001,"15min":0.49399999999999999},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.152},"max":{"1min":0.80600000000000005,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.47599999999999998,"15min":0.51400000000000001},"min":{"1min":0.192,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":2.1139999999999999,"5min":5.1310000000000002,"15min":5.8879999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.47099999999999997,"15min":0.46600000000000003},"min":{"1min":0.21099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":4.4640000000000004,"5min":5.806,"15min":6.0709999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.51400000000000001,"15min":0.51300000000000001},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5099999999999998,"5min":5.843,"15min":6.1059999999999999},"last":0.39300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739349,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5242336,"kb_used_data":4354840,"kb_used_omap":753,"kb_used_meta":886734,"kb_avail":89129504,"statfs":{"total":96636764160,"available":91268612096,"internally_reserved":0,"allocated":4459356160,"data_stored":5224855259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":908016373},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.44900000000000001},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.3279999999999998,"5min":2.7549999999999999,"15min":6.1130000000000004},"last":0.31},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.49399999999999999,"15min":0.54200000000000004},"min":{"1min":0.19700000000000001,"5min":0.191,"15min":0.191},"max":{"1min":6.2320000000000002,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.371}]},{"osd":2,"last update":"Sat Oct 14 01:25:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.42299999999999999,"15min":0.65100000000000002},"min":{"1min":0.22700000000000001,"5min":0.182,"15min":0.17299999999999999},"max":{"1min":1.1870000000000001,"5min":4.6120000000000001,"15min":31.103000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.38,"15min":0.54000000000000004},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.52400000000000002,"5min":4.5419999999999998,"15min":30.234000000000002},"last":0.35399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4458328064,"data_stored":5224513536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4476137472,"data_stored":5243965459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8936243200,"data_stored":10470256659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:25:53.724 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:25:53.724 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:13.726 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:14.606 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:26:14.607 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:26:14.644 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2544,"stamp":"2023-10-14T01:26:14.529127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29881855820,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2218018,"num_read_kb":8674750,"num_write":2191451,"num_write_kb":8764815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88354,"ondisk_log_size":88354,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21360408,"kb_used_data":17524924,"kb_used_omap":2882,"kb_used_meta":3832573,"kb_avail":261755112,"statfs":{"total":289910292480,"available":268037234688,"internally_reserved":0,"allocated":17945522176,"data_stored":21011538092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":3924554765},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,8,12,16,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21200896,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5153,"num_read_kb":20544,"num_write":5207,"num_write_kb":20828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007787"},"pg_stats":[{"pgid":"2.7","version":"99'11342","reported_seq":15594,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:13.100643+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:26:13.100643+0000","last_peered":"2023-10-14T01:26:13.100643+0000","last_clean":"2023-10-14T01:26:13.100643+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:26:13.100643+0000","last_undegraded":"2023-10-14T01:26:13.100643+0000","last_fullsized":"2023-10-14T01:26:13.100643+0000","mapping_epoch":12,"log_start":"55'1300","ondisk_log_start":"55'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7635","reported_seq":11428,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:13.077561+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:26:13.077561+0000","last_peered":"2023-10-14T01:26:13.077561+0000","last_clean":"2023-10-14T01:26:13.077561+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:26:13.077561+0000","last_undegraded":"2023-10-14T01:26:13.077561+0000","last_fullsized":"2023-10-14T01:26:13.077561+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":188,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:24:37.324639+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:24:37.324639+0000","last_peered":"2023-10-14T01:24:37.324639+0000","last_clean":"2023-10-14T01:24:37.324639+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:24:37.324639+0000","last_undegraded":"2023-10-14T01:24:37.324639+0000","last_fullsized":"2023-10-14T01:24:37.324639+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15075","reported_seq":21107,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:13.104111+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:26:13.104111+0000","last_peered":"2023-10-14T01:26:13.104111+0000","last_clean":"2023-10-14T01:26:13.104111+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:26:13.104111+0000","last_undegraded":"2023-10-14T01:26:13.104111+0000","last_fullsized":"2023-10-14T01:26:13.104111+0000","mapping_epoch":12,"log_start":"81'5000","ondisk_log_start":"81'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'545544","reported_seq":1084454,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.398431+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:26:12.398431+0000","last_peered":"2023-10-14T01:26:12.398431+0000","last_clean":"2023-10-14T01:26:12.398431+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:26:12.398431+0000","last_undegraded":"2023-10-14T01:26:12.398431+0000","last_fullsized":"2023-10-14T01:26:12.398431+0000","mapping_epoch":15,"log_start":"97'535534","ondisk_log_start":"97'535534","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7361454080,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":540911,"num_read_kb":2132392,"num_write":540371,"num_write_kb":2161484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11162","reported_seq":15298,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:07.933056+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:26:07.933056+0000","last_peered":"2023-10-14T01:26:07.933056+0000","last_clean":"2023-10-14T01:26:07.933056+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:26:07.933056+0000","last_undegraded":"2023-10-14T01:26:07.933056+0000","last_fullsized":"2023-10-14T01:26:07.933056+0000","mapping_epoch":12,"log_start":"53'1100","ondisk_log_start":"53'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'553950","reported_seq":1101469,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:13.537190+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:26:13.537190+0000","last_peered":"2023-10-14T01:26:13.537190+0000","last_clean":"2023-10-14T01:26:13.537190+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:26:13.537190+0000","last_undegraded":"2023-10-14T01:26:13.537190+0000","last_fullsized":"2023-10-14T01:26:13.537190+0000","mapping_epoch":15,"log_start":"97'543928","ondisk_log_start":"97'543928","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7495720960,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":549559,"num_read_kb":2166300,"num_write":548681,"num_write_kb":2194724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'149","reported_seq":4248,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:07.891389+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:26:07.891389+0000","last_peered":"2023-10-14T01:26:07.891389+0000","last_clean":"2023-10-14T01:26:07.891389+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:26:07.891389+0000","last_undegraded":"2023-10-14T01:26:07.891389+0000","last_fullsized":"2023-10-14T01:26:07.891389+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'557562","reported_seq":1109109,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:12.398596+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:26:12.398596+0000","last_peered":"2023-10-14T01:26:12.398596+0000","last_clean":"2023-10-14T01:26:12.398596+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:26:12.398596+0000","last_undegraded":"2023-10-14T01:26:12.398596+0000","last_fullsized":"2023-10-14T01:26:12.398596+0000","mapping_epoch":15,"log_start":"97'547527","ondisk_log_start":"97'547527","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7521955840,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":553602,"num_read_kb":2183000,"num_write":552279,"num_write_kb":2209116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'69","reported_seq":1540,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:02.797655+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:26:02.797655+0000","last_peered":"2023-10-14T01:26:02.797655+0000","last_clean":"2023-10-14T01:26:02.797655+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:26:02.797655+0000","last_undegraded":"2023-10-14T01:26:02.797655+0000","last_fullsized":"2023-10-14T01:26:02.797655+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'553679","reported_seq":1103209,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:13.547319+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:26:13.547319+0000","last_peered":"2023-10-14T01:26:13.547319+0000","last_clean":"2023-10-14T01:26:13.547319+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:26:13.547319+0000","last_undegraded":"2023-10-14T01:26:13.547319+0000","last_fullsized":"2023-10-14T01:26:13.547319+0000","mapping_epoch":15,"log_start":"98'543635","ondisk_log_start":"98'543635","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2609,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 427s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7502131219,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":551509,"num_read_kb":2174144,"num_write":549310,"num_write_kb":2197234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'14988","reported_seq":20370,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:13.112541+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:26:13.112541+0000","last_peered":"2023-10-14T01:26:13.112541+0000","last_clean":"2023-10-14T01:26:13.112541+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:26:13.112541+0000","last_undegraded":"2023-10-14T01:26:13.112541+0000","last_fullsized":"2023-10-14T01:26:13.112541+0000","mapping_epoch":12,"log_start":"81'4900","ondisk_log_start":"81'4900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":277,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:13.231536+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:13.231536+0000","last_peered":"2023-10-14T01:25:13.231536+0000","last_clean":"2023-10-14T01:25:13.231536+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:13.231536+0000","last_undegraded":"2023-10-14T01:25:13.231536+0000","last_fullsized":"2023-10-14T01:25:13.231536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29881262099,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2195581,"num_read_kb":8655836,"num_write":2190641,"num_write_kb":8762558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17942867968,"data_stored":21010894886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40111,"ondisk_log_size":40111,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48144,"ondisk_log_size":48144,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739350,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10676492,"kb_used_data":8759936,"kb_used_omap":1425,"kb_used_meta":1915118,"kb_avail":83695348,"statfs":{"total":96636764160,"available":85704036352,"internally_reserved":0,"allocated":8970174464,"data_stored":10503642921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":1961081607},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:48 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.40200000000000002,"15min":0.34899999999999998},"min":{"1min":0.21299999999999999,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.54300000000000004,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.35699999999999998,"15min":0.34799999999999998},"min":{"1min":0.23999999999999999,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.1679999999999999,"5min":1.7529999999999999,"15min":3.7519999999999998},"last":0.248}]},{"osd":1,"last update":"Sat Oct 14 01:25:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.51800000000000002,"15min":0.47099999999999997},"min":{"1min":0.23400000000000001,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.782,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":4.3730000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.57499999999999996,"15min":0.52000000000000002},"min":{"1min":0.27400000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.8200000000000001,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":4.4089999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739351,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5381308,"kb_used_data":4393652,"kb_used_omap":704,"kb_used_meta":986943,"kb_avail":88990532,"statfs":{"total":96636764160,"available":91126304768,"internally_reserved":0,"allocated":4499099648,"data_stored":5266148008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1010630161},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,8,8,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.50900000000000001,"15min":0.49399999999999999},"min":{"1min":0.17699999999999999,"5min":0.158,"15min":0.152},"max":{"1min":0.80600000000000005,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.184},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.47599999999999998,"15min":0.51400000000000001},"min":{"1min":0.192,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":2.1139999999999999,"5min":5.1310000000000002,"15min":5.8879999999999999},"last":0.20999999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:25:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.47099999999999997,"15min":0.46600000000000003},"min":{"1min":0.21099999999999999,"5min":0.186,"15min":0.186},"max":{"1min":4.4640000000000004,"5min":5.806,"15min":6.0709999999999997},"last":0.307},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.51400000000000001,"15min":0.51300000000000001},"min":{"1min":0.26200000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.5099999999999998,"5min":5.843,"15min":6.1059999999999999},"last":0.33200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739353,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5302608,"kb_used_data":4371336,"kb_used_omap":753,"kb_used_meta":930510,"kb_avail":89069232,"statfs":{"total":96636764160,"available":91206893568,"internally_reserved":0,"allocated":4476248064,"data_stored":5241747163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":952842997},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.32900000000000001,"15min":0.44},"min":{"1min":0.189,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.53400000000000003,"5min":2.7549999999999999,"15min":6.1130000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":4.5679999999999996,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.27000000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.63500000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.921,"5min":4.6120000000000001,"15min":31.103000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.378,"15min":0.54000000000000004},"min":{"1min":0.222,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.56799999999999995,"5min":4.5419999999999998,"15min":30.234000000000002},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4475502592,"data_stored":5241688064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4497371136,"data_stored":5265199123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8969994240,"data_stored":10504007699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:26:14.645 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:14.646 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:26:14.859 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:14.858+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:26:15.160 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:26:15.189 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:27.034 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:27.029+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:26:27.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:27.045+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:26:27.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:27.045+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:26:27.056 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:27.045+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:26:35.191 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:35.752 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:26:35.753 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:26:35.786 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2554,"stamp":"2023-10-14T01:26:34.532503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29915938636,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2226439,"num_read_kb":8708322,"num_write":2199758,"num_write_kb":8798043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88487,"ondisk_log_size":88487,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21611868,"kb_used_data":17591812,"kb_used_omap":2882,"kb_used_meta":4017149,"kb_avail":261503652,"statfs":{"total":289910292480,"available":267779739648,"internally_reserved":0,"allocated":18014015488,"data_stored":21080031404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":4113560589},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,4,6,10,14,15],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21315584,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5331,"num_read_kb":21268,"num_write":5184,"num_write_kb":20736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001990"},"pg_stats":[{"pgid":"2.7","version":"99'11394","reported_seq":15658,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.490435+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:26:33.490435+0000","last_peered":"2023-10-14T01:26:33.490435+0000","last_clean":"2023-10-14T01:26:33.490435+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:26:33.490435+0000","last_undegraded":"2023-10-14T01:26:33.490435+0000","last_fullsized":"2023-10-14T01:26:33.490435+0000","mapping_epoch":12,"log_start":"55'1300","ondisk_log_start":"55'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7671","reported_seq":11472,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.490234+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:26:33.490234+0000","last_peered":"2023-10-14T01:26:33.490234+0000","last_clean":"2023-10-14T01:26:33.490234+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:26:33.490234+0000","last_undegraded":"2023-10-14T01:26:33.490234+0000","last_fullsized":"2023-10-14T01:26:33.490234+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7671,"log_dups_size":0,"ondisk_log_size":7671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15145","reported_seq":21194,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.490886+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:26:33.490886+0000","last_peered":"2023-10-14T01:26:33.490886+0000","last_clean":"2023-10-14T01:26:33.490886+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:26:33.490886+0000","last_undegraded":"2023-10-14T01:26:33.490886+0000","last_fullsized":"2023-10-14T01:26:33.490886+0000","mapping_epoch":12,"log_start":"81'5100","ondisk_log_start":"81'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'547574","reported_seq":1088464,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:32.427541+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:26:32.427541+0000","last_peered":"2023-10-14T01:26:32.427541+0000","last_clean":"2023-10-14T01:26:32.427541+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:26:32.427541+0000","last_undegraded":"2023-10-14T01:26:32.427541+0000","last_fullsized":"2023-10-14T01:26:32.427541+0000","mapping_epoch":15,"log_start":"98'537534","ondisk_log_start":"98'537534","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7369728000,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":542891,"num_read_kb":2140284,"num_write":542401,"num_write_kb":2169604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11192","reported_seq":15341,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:29.210797+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:26:29.210797+0000","last_peered":"2023-10-14T01:26:29.210797+0000","last_clean":"2023-10-14T01:26:29.210797+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:26:29.210797+0000","last_undegraded":"2023-10-14T01:26:29.210797+0000","last_fullsized":"2023-10-14T01:26:29.210797+0000","mapping_epoch":12,"log_start":"53'1100","ondisk_log_start":"53'1100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'556056","reported_seq":1105730,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.621070+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:26:33.621070+0000","last_peered":"2023-10-14T01:26:33.621070+0000","last_clean":"2023-10-14T01:26:33.621070+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:26:33.621070+0000","last_undegraded":"2023-10-14T01:26:33.621070+0000","last_fullsized":"2023-10-14T01:26:33.621070+0000","mapping_epoch":15,"log_start":"98'546028","ondisk_log_start":"98'546028","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7504281600,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":551708,"num_read_kb":2174864,"num_write":550787,"num_write_kb":2203148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4261,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:28.382513+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:26:28.382513+0000","last_peered":"2023-10-14T01:26:28.382513+0000","last_clean":"2023-10-14T01:26:28.382513+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:26:28.382513+0000","last_undegraded":"2023-10-14T01:26:28.382513+0000","last_fullsized":"2023-10-14T01:26:28.382513+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'559673","reported_seq":1113363,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:32.431770+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:26:32.431770+0000","last_peered":"2023-10-14T01:26:32.431770+0000","last_clean":"2023-10-14T01:26:32.431770+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:26:32.431770+0000","last_undegraded":"2023-10-14T01:26:32.431770+0000","last_fullsized":"2023-10-14T01:26:32.431770+0000","mapping_epoch":15,"log_start":"98'549627","ondisk_log_start":"98'549627","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7530745856,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":555751,"num_read_kb":2191560,"num_write":554390,"num_write_kb":2217560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1544,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:23.253664+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:26:23.253664+0000","last_peered":"2023-10-14T01:26:23.253664+0000","last_clean":"2023-10-14T01:26:23.253664+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:26:23.253664+0000","last_undegraded":"2023-10-14T01:26:23.253664+0000","last_fullsized":"2023-10-14T01:26:23.253664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'555803","reported_seq":1107488,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:33.662625+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:26:33.662625+0000","last_peered":"2023-10-14T01:26:33.662625+0000","last_clean":"2023-10-14T01:26:33.662625+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:26:33.662625+0000","last_undegraded":"2023-10-14T01:26:33.662625+0000","last_fullsized":"2023-10-14T01:26:33.662625+0000","mapping_epoch":15,"log_start":"98'545735","ondisk_log_start":"98'545735","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2699,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 447s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7510589459,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":553652,"num_read_kb":2182700,"num_write":551370,"num_write_kb":2205474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15058","reported_seq":20457,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.490894+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:26:33.490894+0000","last_peered":"2023-10-14T01:26:33.490894+0000","last_clean":"2023-10-14T01:26:33.490894+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:26:33.490894+0000","last_undegraded":"2023-10-14T01:26:33.490894+0000","last_fullsized":"2023-10-14T01:26:33.490894+0000","mapping_epoch":12,"log_start":"81'5000","ondisk_log_start":"81'5000","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":277,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:13.231536+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:13.231536+0000","last_peered":"2023-10-14T01:25:13.231536+0000","last_clean":"2023-10-14T01:25:13.231536+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:13.231536+0000","last_undegraded":"2023-10-14T01:25:13.231536+0000","last_fullsized":"2023-10-14T01:25:13.231536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29915344915,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2204002,"num_read_kb":8689408,"num_write":2198948,"num_write_kb":8795786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18011262976,"data_stored":21079289894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40182,"ondisk_log_size":40182,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48206,"ondisk_log_size":48206,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739354,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10818176,"kb_used_data":8793400,"kb_used_omap":1425,"kb_used_meta":2023342,"kb_avail":83553664,"statfs":{"total":96636764160,"available":85558951936,"internally_reserved":0,"allocated":9004441600,"data_stored":10537910057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":2071902983},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:25:48 2023","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.40200000000000002,"15min":0.34899999999999998},"min":{"1min":0.21299999999999999,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.54300000000000004,"5min":5.1449999999999996,"15min":5.1449999999999996},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.35699999999999998,"15min":0.34799999999999998},"min":{"1min":0.23999999999999999,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":1.1679999999999999,"5min":1.7529999999999999,"15min":3.7519999999999998},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Oct 14 01:25:48 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.51800000000000002,"15min":0.47099999999999997},"min":{"1min":0.23400000000000001,"5min":0.21199999999999999,"15min":0.189},"max":{"1min":1.782,"5min":7.1859999999999999,"15min":7.1859999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.57499999999999996,"15min":0.52000000000000002},"min":{"1min":0.27400000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.8200000000000001,"5min":7.8449999999999998,"15min":7.8449999999999998},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359739355,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5438240,"kb_used_data":4410328,"kb_used_omap":704,"kb_used_meta":1027199,"kb_avail":88933600,"statfs":{"total":96636764160,"available":91068006400,"internally_reserved":0,"allocated":4516175872,"data_stored":5283224232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1051852305},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,3,6,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.58199999999999996,"15min":0.51600000000000001},"min":{"1min":0.182,"5min":0.158,"15min":0.152},"max":{"1min":4.3550000000000004,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.63300000000000001,"15min":0.54900000000000004},"min":{"1min":0.20999999999999999,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":9.2799999999999994,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:26:31 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.61499999999999999,"15min":0.51700000000000002},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":11.965999999999999,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.375},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.70699999999999996,"15min":0.57999999999999996},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":12.000999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.40000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739357,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5355452,"kb_used_data":4388084,"kb_used_omap":753,"kb_used_meta":966606,"kb_avail":89016388,"statfs":{"total":96636764160,"available":91152781312,"internally_reserved":0,"allocated":4493398016,"data_stored":5258897115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":989805301},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,4,5,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.32900000000000001,"15min":0.44},"min":{"1min":0.189,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.53400000000000003,"5min":2.7549999999999999,"15min":6.1130000000000004},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":4.5679999999999996,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.39200000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.63500000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.921,"5min":4.6120000000000001,"15min":31.103000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.378,"15min":0.54000000000000004},"min":{"1min":0.222,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.56799999999999995,"5min":4.5419999999999998,"15min":30.234000000000002},"last":0.29099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4492455936,"data_stored":5258641408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4514455552,"data_stored":5282283539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9004351488,"data_stored":10538364947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:26:35.788 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:35.788 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:26:43.880 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:26:43.877+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:26:49.605 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:49.601+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:26:49.606 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:49.601+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:26:49.606 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:49.601+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:26:51.528 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:26:51.525+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:26:51.534 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:26:51.529+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:26:54.115 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:26:54.109+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:26:55.790 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:26:56.346 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:26:56.347 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:26:56.377 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2564,"stamp":"2023-10-14T01:26:54.535902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29950119756,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2234575,"num_read_kb":8740778,"num_write":2208110,"num_write_kb":8831451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88420,"ondisk_log_size":88420,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21855968,"kb_used_data":17658248,"kb_used_omap":2882,"kb_used_meta":4194813,"kb_avail":261259552,"statfs":{"total":289910292480,"available":267529781248,"internally_reserved":0,"allocated":18082045952,"data_stored":21148061868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":4295488525},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,11,7,7,2,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21319680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5204,"num_read_kb":20744,"num_write":5221,"num_write_kb":20884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001976"},"pg_stats":[{"pgid":"2.7","version":"99'11448","reported_seq":15721,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.127337+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:26:53.127337+0000","last_peered":"2023-10-14T01:26:53.127337+0000","last_clean":"2023-10-14T01:26:53.127337+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:26:53.127337+0000","last_undegraded":"2023-10-14T01:26:53.127337+0000","last_fullsized":"2023-10-14T01:26:53.127337+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7707","reported_seq":11514,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.108809+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:26:53.108809+0000","last_peered":"2023-10-14T01:26:53.108809+0000","last_clean":"2023-10-14T01:26:53.108809+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:26:53.108809+0000","last_undegraded":"2023-10-14T01:26:53.108809+0000","last_fullsized":"2023-10-14T01:26:53.108809+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15217","reported_seq":21278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.127426+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:26:53.127426+0000","last_peered":"2023-10-14T01:26:53.127426+0000","last_clean":"2023-10-14T01:26:53.127426+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:26:53.127426+0000","last_undegraded":"2023-10-14T01:26:53.127426+0000","last_fullsized":"2023-10-14T01:26:53.127426+0000","mapping_epoch":12,"log_start":"81'5200","ondisk_log_start":"81'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'549603","reported_seq":1092446,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:52.505756+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:26:52.505756+0000","last_peered":"2023-10-14T01:26:52.505756+0000","last_clean":"2023-10-14T01:26:52.505756+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:26:52.505756+0000","last_undegraded":"2023-10-14T01:26:52.505756+0000","last_fullsized":"2023-10-14T01:26:52.505756+0000","mapping_epoch":15,"log_start":"98'539534","ondisk_log_start":"98'539534","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7378022400,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":544845,"num_read_kb":2148084,"num_write":544430,"num_write_kb":2177720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11241","reported_seq":15402,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:52.257794+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:26:52.257794+0000","last_peered":"2023-10-14T01:26:52.257794+0000","last_clean":"2023-10-14T01:26:52.257794+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:26:52.257794+0000","last_undegraded":"2023-10-14T01:26:52.257794+0000","last_fullsized":"2023-10-14T01:26:52.257794+0000","mapping_epoch":12,"log_start":"55'1200","ondisk_log_start":"55'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'558150","reported_seq":1109938,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.790591+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:26:53.790591+0000","last_peered":"2023-10-14T01:26:53.790591+0000","last_clean":"2023-10-14T01:26:53.790591+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:26:53.790591+0000","last_undegraded":"2023-10-14T01:26:53.790591+0000","last_fullsized":"2023-10-14T01:26:53.790591+0000","mapping_epoch":15,"log_start":"98'548128","ondisk_log_start":"98'548128","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7512817664,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":553823,"num_read_kb":2183300,"num_write":552881,"num_write_kb":2211524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4273,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:48.853925+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:26:48.853925+0000","last_peered":"2023-10-14T01:26:48.853925+0000","last_clean":"2023-10-14T01:26:48.853925+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:26:48.853925+0000","last_undegraded":"2023-10-14T01:26:48.853925+0000","last_fullsized":"2023-10-14T01:26:48.853925+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'561783","reported_seq":1117580,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:52.519306+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:26:52.519306+0000","last_peered":"2023-10-14T01:26:52.519306+0000","last_clean":"2023-10-14T01:26:52.519306+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:26:52.519306+0000","last_undegraded":"2023-10-14T01:26:52.519306+0000","last_fullsized":"2023-10-14T01:26:52.519306+0000","mapping_epoch":15,"log_start":"98'551727","ondisk_log_start":"98'551727","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7539343360,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":557850,"num_read_kb":2199924,"num_write":556500,"num_write_kb":2226000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1548,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:43.733405+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:26:43.733405+0000","last_peered":"2023-10-14T01:26:43.733405+0000","last_clean":"2023-10-14T01:26:43.733405+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:26:43.733405+0000","last_undegraded":"2023-10-14T01:26:43.733405+0000","last_fullsized":"2023-10-14T01:26:43.733405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'558020","reported_seq":1111684,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:26:53.803618+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:26:53.803618+0000","last_peered":"2023-10-14T01:26:53.803618+0000","last_clean":"2023-10-14T01:26:53.803618+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:26:53.803618+0000","last_undegraded":"2023-10-14T01:26:53.803618+0000","last_fullsized":"2023-10-14T01:26:53.803618+0000","mapping_epoch":15,"log_start":"98'547935","ondisk_log_start":"98'547935","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2840,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 467s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7519342611,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":555620,"num_read_kb":2190556,"num_write":553489,"num_write_kb":2213950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15130","reported_seq":20541,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:53.172949+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:26:53.172949+0000","last_peered":"2023-10-14T01:26:53.172949+0000","last_clean":"2023-10-14T01:26:53.172949+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:26:53.172949+0000","last_undegraded":"2023-10-14T01:26:53.172949+0000","last_fullsized":"2023-10-14T01:26:53.172949+0000","mapping_epoch":12,"log_start":"81'5100","ondisk_log_start":"81'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":277,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-10-14T01:25:13.231536+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:13.231536+0000","last_peered":"2023-10-14T01:25:13.231536+0000","last_clean":"2023-10-14T01:25:13.231536+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:13.231536+0000","last_undegraded":"2023-10-14T01:25:13.231536+0000","last_fullsized":"2023-10-14T01:25:13.231536+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29949526035,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2212138,"num_read_kb":8721864,"num_write":2207300,"num_write_kb":8829194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18079588352,"data_stored":21147615270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40232,"ondisk_log_size":40232,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48089,"ondisk_log_size":48089,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739358,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10937416,"kb_used_data":8826560,"kb_used_omap":1425,"kb_used_meta":2109422,"kb_avail":83434424,"statfs":{"total":96636764160,"available":85436850176,"internally_reserved":0,"allocated":9038397440,"data_stored":10571865897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":2160048903},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:48 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.33200000000000002,"15min":0.34899999999999998},"min":{"1min":0.20599999999999999,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":0.58499999999999996,"5min":3.1099999999999999,"15min":5.1449999999999996},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.42299999999999999,"15min":0.36899999999999999},"min":{"1min":0.191,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":3.835,"5min":3.835,"15min":3.835},"last":0.247}]},{"osd":1,"last update":"Sat Oct 14 01:26:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.53700000000000003,"15min":0.50600000000000001},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.19800000000000001},"max":{"1min":5.468,"5min":5.468,"15min":7.1859999999999999},"last":1.7330000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.58799999999999997,"15min":0.55600000000000005},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":5.5019999999999998,"5min":5.5019999999999998,"15min":7.8449999999999998},"last":1.8069999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739359,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5510452,"kb_used_data":4427052,"kb_used_omap":704,"kb_used_meta":1082687,"kb_avail":88861388,"statfs":{"total":96636764160,"available":90994061312,"internally_reserved":0,"allocated":4533301248,"data_stored":5300349608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1108672017},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1,3,2,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.58199999999999996,"15min":0.51600000000000001},"min":{"1min":0.182,"5min":0.158,"15min":0.152},"max":{"1min":4.3550000000000004,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.17599999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.63300000000000001,"15min":0.54900000000000004},"min":{"1min":0.20999999999999999,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":9.2799999999999994,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.25900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:26:31 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.61499999999999999,"15min":0.51700000000000002},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":11.965999999999999,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.70699999999999996,"15min":0.57999999999999996},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":12.000999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.36799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739361,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5408100,"kb_used_data":4404636,"kb_used_omap":753,"kb_used_meta":1002702,"kb_avail":88963740,"statfs":{"total":96636764160,"available":91098869760,"internally_reserved":0,"allocated":4510347264,"data_stored":5275846363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1026767605},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.32900000000000001,"15min":0.44},"min":{"1min":0.189,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.53400000000000003,"5min":2.7549999999999999,"15min":6.1130000000000004},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":4.5679999999999996,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":2.0630000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.63500000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.921,"5min":4.6120000000000001,"15min":31.103000000000002},"last":1.8740000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.378,"15min":0.54000000000000004},"min":{"1min":0.222,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.56799999999999995,"5min":4.5419999999999998,"15min":30.234000000000002},"last":0.30499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4509429760,"data_stored":5275615232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4531695616,"data_stored":5299523603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9038462976,"data_stored":10572476435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:26:56.379 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:26:56.380 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:16.381 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:16.877 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:27:16.878 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:27:16.903 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2575,"stamp":"2023-10-14T01:27:16.539588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29985099596,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2242917,"num_read_kb":8774082,"num_write":2216651,"num_write_kb":8865615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88623,"ondisk_log_size":88623,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21968420,"kb_used_data":17726220,"kb_used_omap":2882,"kb_used_meta":4239293,"kb_avail":261147100,"statfs":{"total":289910292480,"available":267414630400,"internally_reserved":0,"allocated":18151649280,"data_stored":21217665196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2952179,"internal_metadata":4341036045},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,4,10,15,16,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17719296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4073,"num_read_kb":16264,"num_write":4330,"num_write_kb":17320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001998"},"pg_stats":[{"pgid":"2.7","version":"99'11487","reported_seq":15772,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.847312+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:27:13.847312+0000","last_peered":"2023-10-14T01:27:13.847312+0000","last_clean":"2023-10-14T01:27:13.847312+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:27:13.847312+0000","last_undegraded":"2023-10-14T01:27:13.847312+0000","last_fullsized":"2023-10-14T01:27:13.847312+0000","mapping_epoch":12,"log_start":"55'1400","ondisk_log_start":"55'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7734","reported_seq":11549,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.817935+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:27:13.817935+0000","last_peered":"2023-10-14T01:27:13.817935+0000","last_clean":"2023-10-14T01:27:13.817935+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:27:13.817935+0000","last_undegraded":"2023-10-14T01:27:13.817935+0000","last_fullsized":"2023-10-14T01:27:13.817935+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7734,"log_dups_size":0,"ondisk_log_size":7734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15269","reported_seq":21346,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.850913+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:27:13.850913+0000","last_peered":"2023-10-14T01:27:13.850913+0000","last_clean":"2023-10-14T01:27:13.850913+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:27:13.850913+0000","last_undegraded":"2023-10-14T01:27:13.850913+0000","last_fullsized":"2023-10-14T01:27:13.850913+0000","mapping_epoch":12,"log_start":"81'5200","ondisk_log_start":"81'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'551704","reported_seq":1096597,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:12.607573+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:27:12.607573+0000","last_peered":"2023-10-14T01:27:12.607573+0000","last_clean":"2023-10-14T01:27:12.607573+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:27:12.607573+0000","last_undegraded":"2023-10-14T01:27:12.607573+0000","last_fullsized":"2023-10-14T01:27:12.607573+0000","mapping_epoch":15,"log_start":"98'541634","ondisk_log_start":"98'541634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7386722304,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":546894,"num_read_kb":2156272,"num_write":546531,"num_write_kb":2186124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11274","reported_seq":15447,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:09.333603+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:27:09.333603+0000","last_peered":"2023-10-14T01:27:09.333603+0000","last_clean":"2023-10-14T01:27:09.333603+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:27:09.333603+0000","last_undegraded":"2023-10-14T01:27:09.333603+0000","last_fullsized":"2023-10-14T01:27:09.333603+0000","mapping_epoch":12,"log_start":"55'1200","ondisk_log_start":"55'1200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'560348","reported_seq":1114232,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.908902+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:27:13.908902+0000","last_peered":"2023-10-14T01:27:13.908902+0000","last_clean":"2023-10-14T01:27:13.908902+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:27:13.908902+0000","last_undegraded":"2023-10-14T01:27:13.908902+0000","last_fullsized":"2023-10-14T01:27:13.908902+0000","mapping_epoch":15,"log_start":"98'550328","ondisk_log_start":"98'550328","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7521800192,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":555919,"num_read_kb":2191656,"num_write":555079,"num_write_kb":2220316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4285,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:09.353854+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:27:09.353854+0000","last_peered":"2023-10-14T01:27:09.353854+0000","last_clean":"2023-10-14T01:27:09.353854+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:27:09.353854+0000","last_undegraded":"2023-10-14T01:27:09.353854+0000","last_fullsized":"2023-10-14T01:27:09.353854+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'563971","reported_seq":1121882,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:12.599160+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:27:12.599160+0000","last_peered":"2023-10-14T01:27:12.599160+0000","last_clean":"2023-10-14T01:27:12.599160+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:27:12.599160+0000","last_undegraded":"2023-10-14T01:27:12.599160+0000","last_fullsized":"2023-10-14T01:27:12.599160+0000","mapping_epoch":15,"log_start":"98'553927","ondisk_log_start":"98'553927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7548289024,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":559968,"num_read_kb":2208384,"num_write":558688,"num_write_kb":2234752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1552,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:04.210079+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:27:04.210079+0000","last_peered":"2023-10-14T01:27:04.210079+0000","last_clean":"2023-10-14T01:27:04.210079+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:27:04.210079+0000","last_undegraded":"2023-10-14T01:27:04.210079+0000","last_fullsized":"2023-10-14T01:27:04.210079+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'560132","reported_seq":1115884,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:13.919955+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:27:13.919955+0000","last_peered":"2023-10-14T01:27:13.919955+0000","last_clean":"2023-10-14T01:27:13.919955+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:27:13.919955+0000","last_undegraded":"2023-10-14T01:27:13.919955+0000","last_fullsized":"2023-10-14T01:27:13.919955+0000","mapping_epoch":15,"log_start":"98'550035","ondisk_log_start":"98'550035","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":2924,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 487s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7527694355,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":557699,"num_read_kb":2198856,"num_write":555543,"num_write_kb":2222166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15183","reported_seq":20610,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.856091+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:27:13.856091+0000","last_peered":"2023-10-14T01:27:13.856091+0000","last_clean":"2023-10-14T01:27:13.856091+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:27:13.856091+0000","last_undegraded":"2023-10-14T01:27:13.856091+0000","last_fullsized":"2023-10-14T01:27:13.856091+0000","mapping_epoch":12,"log_start":"81'5100","ondisk_log_start":"81'5100","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343110+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:47.343110+0000","last_peered":"2023-10-14T01:25:47.343110+0000","last_clean":"2023-10-14T01:25:47.343110+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:47.343110+0000","last_undegraded":"2023-10-14T01:25:47.343110+0000","last_fullsized":"2023-10-14T01:25:47.343110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":29984505875,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2220480,"num_read_kb":8755168,"num_write":2215841,"num_write_kb":8863358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18149330944,"data_stored":21217357862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40231,"ondisk_log_size":40231,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48293,"ondisk_log_size":48293,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739362,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10939480,"kb_used_data":8860496,"kb_used_omap":1425,"kb_used_meta":2077550,"kb_avail":83432360,"statfs":{"total":96636764160,"available":85434736640,"internally_reserved":0,"allocated":9073147904,"data_stored":10606616361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1459449,"internal_metadata":2127411975},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:48 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.33200000000000002,"15min":0.34899999999999998},"min":{"1min":0.20599999999999999,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":0.58499999999999996,"5min":3.1099999999999999,"15min":5.1449999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.42299999999999999,"15min":0.36899999999999999},"min":{"1min":0.191,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":3.835,"5min":3.835,"15min":3.835},"last":0.435}]},{"osd":1,"last update":"Sat Oct 14 01:26:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.53700000000000003,"15min":0.50600000000000001},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.19800000000000001},"max":{"1min":5.468,"5min":5.468,"15min":7.1859999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.58799999999999997,"15min":0.55600000000000005},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":5.5019999999999998,"5min":5.5019999999999998,"15min":7.8449999999999998},"last":0.52000000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739363,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5563528,"kb_used_data":4444032,"kb_used_omap":704,"kb_used_meta":1118783,"kb_avail":88808312,"statfs":{"total":96636764160,"available":90939711488,"internally_reserved":0,"allocated":4550688768,"data_stored":5317737128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1145634321},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,5,6,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.58199999999999996,"15min":0.51600000000000001},"min":{"1min":0.182,"5min":0.158,"15min":0.152},"max":{"1min":4.3550000000000004,"5min":6.0019999999999998,"15min":7.2809999999999997},"last":0.219},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.63300000000000001,"15min":0.54900000000000004},"min":{"1min":0.20999999999999999,"5min":0.18099999999999999,"15min":0.17799999999999999},"max":{"1min":9.2799999999999994,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:26:31 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":0.61499999999999999,"15min":0.51700000000000002},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":11.965999999999999,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.4179999999999999,"5min":0.70699999999999996,"15min":0.57999999999999996},"min":{"1min":0.251,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":12.000999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.37}]}]},{"osd":0,"up_from":8,"seq":34359739365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5465412,"kb_used_data":4421692,"kb_used_omap":753,"kb_used_meta":1042958,"kb_avail":88906428,"statfs":{"total":96636764160,"available":91040182272,"internally_reserved":0,"allocated":4527812608,"data_stored":5293311707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1067989749},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,4,9,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.32900000000000001,"15min":0.44},"min":{"1min":0.189,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.53400000000000003,"5min":2.7549999999999999,"15min":6.1130000000000004},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.51600000000000001,"15min":0.54300000000000004},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":4.5679999999999996,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:26:11 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.63500000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.921,"5min":4.6120000000000001,"15min":31.103000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.378,"15min":0.54000000000000004},"min":{"1min":0.222,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.56799999999999995,"5min":4.5419999999999998,"15min":30.234000000000002},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4526985216,"data_stored":5293170688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4549107712,"data_stored":5316935699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9073238016,"data_stored":10607251475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:27:16.904 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:16.905 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:36.906 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:37.450 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:27:37.452 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:27:37.483 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2585,"stamp":"2023-10-14T01:27:36.542837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30019342156,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2251288,"num_read_kb":8807470,"num_write":2225034,"num_write_kb":8899147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88374,"ondisk_log_size":88374,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21426060,"kb_used_data":17793204,"kb_used_omap":2884,"kb_used_meta":3629947,"kb_avail":261689460,"statfs":{"total":289910292480,"available":267970007040,"internally_reserved":0,"allocated":18220240896,"data_stored":21286256812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":3717066109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,5,13,12,12,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17154048,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4222,"num_read_kb":16848,"num_write":4197,"num_write_kb":16788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001941"},"pg_stats":[{"pgid":"2.7","version":"99'11514","reported_seq":15812,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.316189+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:27:31.316189+0000","last_peered":"2023-10-14T01:27:31.316189+0000","last_clean":"2023-10-14T01:27:31.316189+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:27:31.316189+0000","last_undegraded":"2023-10-14T01:27:31.316189+0000","last_fullsized":"2023-10-14T01:27:31.316189+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7752","reported_seq":11576,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.293498+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:27:31.293498+0000","last_peered":"2023-10-14T01:27:31.293498+0000","last_clean":"2023-10-14T01:27:31.293498+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:27:31.293498+0000","last_undegraded":"2023-10-14T01:27:31.293498+0000","last_fullsized":"2023-10-14T01:27:31.293498+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7752,"log_dups_size":0,"ondisk_log_size":7752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15305","reported_seq":21398,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.316205+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:27:31.316205+0000","last_peered":"2023-10-14T01:27:31.316205+0000","last_clean":"2023-10-14T01:27:31.316205+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:27:31.316205+0000","last_undegraded":"2023-10-14T01:27:31.316205+0000","last_fullsized":"2023-10-14T01:27:31.316205+0000","mapping_epoch":12,"log_start":"81'5300","ondisk_log_start":"81'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'553779","reported_seq":1100778,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.685883+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:27:32.685883+0000","last_peered":"2023-10-14T01:27:32.685883+0000","last_clean":"2023-10-14T01:27:32.685883+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:27:32.685883+0000","last_undegraded":"2023-10-14T01:27:32.685883+0000","last_fullsized":"2023-10-14T01:27:32.685883+0000","mapping_epoch":15,"log_start":"99'543734","ondisk_log_start":"99'543734","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7395221504,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":548999,"num_read_kb":2164676,"num_write":548606,"num_write_kb":2194424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11343","reported_seq":15528,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.004588+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:27:32.004588+0000","last_peered":"2023-10-14T01:27:32.004588+0000","last_clean":"2023-10-14T01:27:32.004588+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:27:32.004588+0000","last_undegraded":"2023-10-14T01:27:32.004588+0000","last_fullsized":"2023-10-14T01:27:32.004588+0000","mapping_epoch":12,"log_start":"55'1300","ondisk_log_start":"55'1300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'562488","reported_seq":1118449,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:33.998892+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:27:33.998892+0000","last_peered":"2023-10-14T01:27:33.998892+0000","last_clean":"2023-10-14T01:27:33.998892+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:27:33.998892+0000","last_undegraded":"2023-10-14T01:27:33.998892+0000","last_fullsized":"2023-10-14T01:27:33.998892+0000","mapping_epoch":15,"log_start":"99'552428","ondisk_log_start":"99'552428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7530463232,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":558000,"num_read_kb":2199956,"num_write":557219,"num_write_kb":2228876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4297,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:29.809778+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:27:29.809778+0000","last_peered":"2023-10-14T01:27:29.809778+0000","last_clean":"2023-10-14T01:27:29.809778+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:27:29.809778+0000","last_undegraded":"2023-10-14T01:27:29.809778+0000","last_fullsized":"2023-10-14T01:27:29.809778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'566032","reported_seq":1126079,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:32.676789+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:27:32.676789+0000","last_peered":"2023-10-14T01:27:32.676789+0000","last_clean":"2023-10-14T01:27:32.676789+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:27:32.676789+0000","last_undegraded":"2023-10-14T01:27:32.676789+0000","last_fullsized":"2023-10-14T01:27:32.676789+0000","mapping_epoch":15,"log_start":"99'556027","ondisk_log_start":"99'556027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7556747264,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":562101,"num_read_kb":2216880,"num_write":560749,"num_write_kb":2242996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1557,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:24.689952+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:27:24.689952+0000","last_peered":"2023-10-14T01:27:24.689952+0000","last_clean":"2023-10-14T01:27:24.689952+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:27:24.689952+0000","last_undegraded":"2023-10-14T01:27:24.689952+0000","last_fullsized":"2023-10-14T01:27:24.689952+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'562321","reported_seq":1120129,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:34.010958+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:27:34.010958+0000","last_peered":"2023-10-14T01:27:34.010958+0000","last_clean":"2023-10-14T01:27:34.010958+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:27:34.010958+0000","last_undegraded":"2023-10-14T01:27:34.010958+0000","last_fullsized":"2023-10-14T01:27:34.010958+0000","mapping_epoch":15,"log_start":"99'552235","ondisk_log_start":"99'552235","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3041,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 507s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7536316435,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":559751,"num_read_kb":2207044,"num_write":557650,"num_write_kb":2230594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15219","reported_seq":20662,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:31.355470+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:27:31.355470+0000","last_peered":"2023-10-14T01:27:31.355470+0000","last_clean":"2023-10-14T01:27:31.355470+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:27:31.355470+0000","last_undegraded":"2023-10-14T01:27:31.355470+0000","last_fullsized":"2023-10-14T01:27:31.355470+0000","mapping_epoch":12,"log_start":"81'5200","ondisk_log_start":"81'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343110+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:47.343110+0000","last_peered":"2023-10-14T01:25:47.343110+0000","last_clean":"2023-10-14T01:25:47.343110+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:47.343110+0000","last_undegraded":"2023-10-14T01:25:47.343110+0000","last_fullsized":"2023-10-14T01:25:47.343110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30018748435,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2228851,"num_read_kb":8788556,"num_write":2224224,"num_write_kb":8896890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18218016768,"data_stored":21286043686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40196,"ondisk_log_size":40196,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48079,"ondisk_log_size":48079,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739366,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10269200,"kb_used_data":8893960,"kb_used_omap":1426,"kb_used_meta":1373805,"kb_avail":84102640,"statfs":{"total":96636764160,"available":86121103360,"internally_reserved":0,"allocated":9107415040,"data_stored":10640883497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":1406776439},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,6,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:26:48 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.33200000000000002,"15min":0.34899999999999998},"min":{"1min":0.20599999999999999,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":0.58499999999999996,"5min":3.1099999999999999,"15min":5.1449999999999996},"last":7.1040000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.42299999999999999,"15min":0.36899999999999999},"min":{"1min":0.191,"5min":0.17299999999999999,"15min":0.16600000000000001},"max":{"1min":3.835,"5min":3.835,"15min":3.835},"last":0.22500000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:26:48 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.53700000000000003,"15min":0.50600000000000001},"min":{"1min":0.24099999999999999,"5min":0.224,"15min":0.19800000000000001},"max":{"1min":5.468,"5min":5.468,"15min":7.1859999999999999},"last":7.1319999999999997},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.58799999999999997,"15min":0.55600000000000005},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":5.5019999999999998,"5min":5.5019999999999998,"15min":7.8449999999999998},"last":7.1479999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739367,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5620684,"kb_used_data":4460932,"kb_used_omap":704,"kb_used_meta":1159039,"kb_avail":88751156,"statfs":{"total":96636764160,"available":90881183744,"internally_reserved":0,"allocated":4567994368,"data_stored":5335042728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1186856465},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,3,6,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:32 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.17599999999999999,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":2.0379999999999998,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.192},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.56799999999999995,"15min":0.55900000000000005},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.1190000000000002,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.24099999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:27:32 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.55800000000000005,"15min":0.51200000000000001},"min":{"1min":0.22900000000000001,"5min":0.187,"15min":0.186},"max":{"1min":0.495,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.64900000000000002,"15min":0.57499999999999996},"min":{"1min":0.26900000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51900000000000002,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.28799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739369,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5536176,"kb_used_data":4438312,"kb_used_omap":753,"kb_used_meta":1097102,"kb_avail":88835664,"statfs":{"total":96636764160,"available":90967719936,"internally_reserved":0,"allocated":4544831488,"data_stored":5310330587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1123433205},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:16 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.42999999999999999},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.5259999999999998,"5min":2.7549999999999999,"15min":5.8600000000000003},"last":0.378},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.22,"5min":0.187,"15min":0.187},"max":{"1min":2.625,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Oct 14 01:27:16 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.63400000000000001},"min":{"1min":0.19700000000000001,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.8740000000000001,"5min":1.921,"15min":31.103000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.54100000000000004},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.55900000000000005,"5min":0.56799999999999995,"15min":30.234000000000002},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4543926272,"data_stored":5310111744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4566503424,"data_stored":5334331411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9107587072,"data_stored":10641600531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:27:37.485 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:37.486 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:27:57.487 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:27:57.973 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:27:57.973 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:27:58.017 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2595,"stamp":"2023-10-14T01:27:56.549171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30053945164,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2259666,"num_read_kb":8840858,"num_write":2233533,"num_write_kb":8933143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88369,"ondisk_log_size":88369,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21738788,"kb_used_data":17861580,"kb_used_omap":2884,"kb_used_meta":3874299,"kb_avail":261376732,"statfs":{"total":289910292480,"available":267649773568,"internally_reserved":0,"allocated":18290257920,"data_stored":21356273836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":3967282557},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,9,41,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17457152,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4308,"num_read_kb":17168,"num_write":4279,"num_write_kb":17116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004809"},"pg_stats":[{"pgid":"2.7","version":"99'11556","reported_seq":15866,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.308969+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:27:52.308969+0000","last_peered":"2023-10-14T01:27:52.308969+0000","last_clean":"2023-10-14T01:27:52.308969+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:27:52.308969+0000","last_undegraded":"2023-10-14T01:27:52.308969+0000","last_fullsized":"2023-10-14T01:27:52.308969+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7780","reported_seq":11612,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.285505+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:27:52.285505+0000","last_peered":"2023-10-14T01:27:52.285505+0000","last_clean":"2023-10-14T01:27:52.285505+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:27:52.285505+0000","last_undegraded":"2023-10-14T01:27:52.285505+0000","last_fullsized":"2023-10-14T01:27:52.285505+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15361","reported_seq":21470,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.307673+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:27:52.307673+0000","last_peered":"2023-10-14T01:27:52.307673+0000","last_clean":"2023-10-14T01:27:52.307673+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:27:52.307673+0000","last_undegraded":"2023-10-14T01:27:52.307673+0000","last_fullsized":"2023-10-14T01:27:52.307673+0000","mapping_epoch":12,"log_start":"81'5300","ondisk_log_start":"81'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'555847","reported_seq":1104915,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.849650+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:27:52.849650+0000","last_peered":"2023-10-14T01:27:52.849650+0000","last_clean":"2023-10-14T01:27:52.849650+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:27:52.849650+0000","last_undegraded":"2023-10-14T01:27:52.849650+0000","last_fullsized":"2023-10-14T01:27:52.849650+0000","mapping_epoch":15,"log_start":"99'545834","ondisk_log_start":"99'545834","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7403601920,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":551070,"num_read_kb":2172932,"num_write":550674,"num_write_kb":2202696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11403","reported_seq":15600,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.329168+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:27:52.329168+0000","last_peered":"2023-10-14T01:27:52.329168+0000","last_clean":"2023-10-14T01:27:52.329168+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:27:52.329168+0000","last_undegraded":"2023-10-14T01:27:52.329168+0000","last_fullsized":"2023-10-14T01:27:52.329168+0000","mapping_epoch":12,"log_start":"57'1400","ondisk_log_start":"57'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'564636","reported_seq":1122708,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:54.115022+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:27:54.115022+0000","last_peered":"2023-10-14T01:27:54.115022+0000","last_clean":"2023-10-14T01:27:54.115022+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:27:54.115022+0000","last_undegraded":"2023-10-14T01:27:54.115022+0000","last_fullsized":"2023-10-14T01:27:54.115022+0000","mapping_epoch":15,"log_start":"99'554628","ondisk_log_start":"99'554628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7539191808,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":560106,"num_read_kb":2208352,"num_write":559367,"num_write_kb":2237468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4309,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:50.294402+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:27:50.294402+0000","last_peered":"2023-10-14T01:27:50.294402+0000","last_clean":"2023-10-14T01:27:50.294402+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:27:50.294402+0000","last_undegraded":"2023-10-14T01:27:50.294402+0000","last_fullsized":"2023-10-14T01:27:50.294402+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'568151","reported_seq":1130291,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.819325+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:27:52.819325+0000","last_peered":"2023-10-14T01:27:52.819325+0000","last_clean":"2023-10-14T01:27:52.819325+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:27:52.819325+0000","last_undegraded":"2023-10-14T01:27:52.819325+0000","last_fullsized":"2023-10-14T01:27:52.819325+0000","mapping_epoch":15,"log_start":"99'558127","ondisk_log_start":"99'558127","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7565393920,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":564194,"num_read_kb":2225228,"num_write":562868,"num_write_kb":2251472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1561,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:45.171398+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:27:45.171398+0000","last_peered":"2023-10-14T01:27:45.171398+0000","last_clean":"2023-10-14T01:27:45.171398+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:27:45.171398+0000","last_undegraded":"2023-10-14T01:27:45.171398+0000","last_fullsized":"2023-10-14T01:27:45.171398+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'564539","reported_seq":1124473,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:27:54.121719+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:27:54.121719+0000","last_peered":"2023-10-14T01:27:54.121719+0000","last_clean":"2023-10-14T01:27:54.121719+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:27:54.121719+0000","last_undegraded":"2023-10-14T01:27:54.121719+0000","last_fullsized":"2023-10-14T01:27:54.121719+0000","mapping_epoch":15,"log_start":"99'554535","ondisk_log_start":"99'554535","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3119,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 528s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7545163795,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":561859,"num_read_kb":2215432,"num_write":559814,"num_write_kb":2239250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15275","reported_seq":20734,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:52.333507+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:27:52.333507+0000","last_peered":"2023-10-14T01:27:52.333507+0000","last_clean":"2023-10-14T01:27:52.333507+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:27:52.333507+0000","last_undegraded":"2023-10-14T01:27:52.333507+0000","last_fullsized":"2023-10-14T01:27:52.333507+0000","mapping_epoch":12,"log_start":"81'5200","ondisk_log_start":"81'5200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343110+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:47.343110+0000","last_peered":"2023-10-14T01:25:47.343110+0000","last_clean":"2023-10-14T01:25:47.343110+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:47.343110+0000","last_undegraded":"2023-10-14T01:25:47.343110+0000","last_fullsized":"2023-10-14T01:25:47.343110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30053351443,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2237229,"num_read_kb":8821944,"num_write":2232723,"num_write_kb":8930886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18287583232,"data_stored":21355610150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40049,"ondisk_log_size":40049,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48221,"ondisk_log_size":48221,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739370,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10457072,"kb_used_data":8927720,"kb_used_omap":1426,"kb_used_meta":1527917,"kb_avail":83914768,"statfs":{"total":96636764160,"available":85928722432,"internally_reserved":0,"allocated":9141985280,"data_stored":10675453737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":1564587127},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.19500000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":7.1040000000000001,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.375},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.16600000000000001},"max":{"1min":0.82599999999999996,"5min":3.835,"15min":3.835},"last":0.20799999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.61499999999999999,"15min":0.52200000000000002},"min":{"1min":0.23699999999999999,"5min":0.224,"15min":0.19800000000000001},"max":{"1min":7.1319999999999997,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66500000000000004,"15min":0.57199999999999995},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":7.1479999999999997,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.29199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739371,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5673984,"kb_used_data":4478136,"kb_used_omap":704,"kb_used_meta":1195135,"kb_avail":88697856,"statfs":{"total":96636764160,"available":90826604544,"internally_reserved":0,"allocated":4585611264,"data_stored":5352659624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1223818769},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,34,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:32 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.17599999999999999,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":2.0379999999999998,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.56799999999999995,"15min":0.55900000000000005},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.1190000000000002,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":1.4339999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:27:32 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.55800000000000005,"15min":0.51200000000000001},"min":{"1min":0.22900000000000001,"5min":0.187,"15min":0.186},"max":{"1min":0.495,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.64900000000000002,"15min":0.57499999999999996},"min":{"1min":0.26900000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51900000000000002,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.47799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739373,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5607732,"kb_used_data":4455724,"kb_used_omap":753,"kb_used_meta":1151246,"kb_avail":88764108,"statfs":{"total":96636764160,"available":90894446592,"internally_reserved":0,"allocated":4562661376,"data_stored":5328160475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1178876661},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,15,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:16 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.42999999999999999},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.5259999999999998,"5min":2.7549999999999999,"15min":5.8600000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.22,"5min":0.187,"15min":0.187},"max":{"1min":2.625,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:27:16 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.63400000000000001},"min":{"1min":0.19700000000000001,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.8740000000000001,"5min":1.921,"15min":31.103000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.54100000000000004},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.55900000000000005,"5min":0.56799999999999995,"15min":30.234000000000002},"last":0.30099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4561072128,"data_stored":5327257600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4584169472,"data_stored":5351997459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9142341632,"data_stored":10676355091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:27:58.019 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:27:58.020 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:18.021 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:18.524 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:28:18.526 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:28:18.551 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2605,"stamp":"2023-10-14T01:28:16.552523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30088322892,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2268122,"num_read_kb":8874574,"num_write":2241946,"num_write_kb":8966795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88513,"ondisk_log_size":88513,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":21970732,"kb_used_data":17929044,"kb_used_omap":2884,"kb_used_meta":4038779,"kb_avail":261144788,"statfs":{"total":289910292480,"available":267412262912,"internally_reserved":0,"allocated":18359341056,"data_stored":21425356972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":4135710077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,4,11,11,10,14,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17379328,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4234,"num_read_kb":16872,"num_write":4251,"num_write_kb":17004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002002"},"pg_stats":[{"pgid":"2.7","version":"99'11604","reported_seq":15926,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:14.076282+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:28:14.076282+0000","last_peered":"2023-10-14T01:28:14.076282+0000","last_clean":"2023-10-14T01:28:14.076282+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:28:14.076282+0000","last_undegraded":"2023-10-14T01:28:14.076282+0000","last_fullsized":"2023-10-14T01:28:14.076282+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7812","reported_seq":11652,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:14.069475+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:28:14.069475+0000","last_peered":"2023-10-14T01:28:14.069475+0000","last_clean":"2023-10-14T01:28:14.069475+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:28:14.069475+0000","last_undegraded":"2023-10-14T01:28:14.069475+0000","last_fullsized":"2023-10-14T01:28:14.069475+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7812,"log_dups_size":0,"ondisk_log_size":7812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15425","reported_seq":21550,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:14.082500+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:28:14.082500+0000","last_peered":"2023-10-14T01:28:14.082500+0000","last_clean":"2023-10-14T01:28:14.082500+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:28:14.082500+0000","last_undegraded":"2023-10-14T01:28:14.082500+0000","last_fullsized":"2023-10-14T01:28:14.082500+0000","mapping_epoch":12,"log_start":"81'5400","ondisk_log_start":"81'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'557904","reported_seq":1109015,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:12.923971+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:28:12.923971+0000","last_peered":"2023-10-14T01:28:12.923971+0000","last_clean":"2023-10-14T01:28:12.923971+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:28:12.923971+0000","last_undegraded":"2023-10-14T01:28:12.923971+0000","last_fullsized":"2023-10-14T01:28:12.923971+0000","mapping_epoch":15,"log_start":"99'547834","ondisk_log_start":"99'547834","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7412031488,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":553111,"num_read_kb":2181080,"num_write":552731,"num_write_kb":2210924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11454","reported_seq":15663,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:12.456872+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:28:12.456872+0000","last_peered":"2023-10-14T01:28:12.456872+0000","last_clean":"2023-10-14T01:28:12.456872+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:28:12.456872+0000","last_undegraded":"2023-10-14T01:28:12.456872+0000","last_fullsized":"2023-10-14T01:28:12.456872+0000","mapping_epoch":12,"log_start":"57'1400","ondisk_log_start":"57'1400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'566724","reported_seq":1126935,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:14.200087+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:28:14.200087+0000","last_peered":"2023-10-14T01:28:14.200087+0000","last_clean":"2023-10-14T01:28:14.200087+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:28:14.200087+0000","last_undegraded":"2023-10-14T01:28:14.200087+0000","last_fullsized":"2023-10-14T01:28:14.200087+0000","mapping_epoch":15,"log_start":"99'556628","ondisk_log_start":"99'556628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7547854848,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":562243,"num_read_kb":2216876,"num_write":561455,"num_write_kb":2245820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4322,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:10.770437+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:28:10.770437+0000","last_peered":"2023-10-14T01:28:10.770437+0000","last_clean":"2023-10-14T01:28:10.770437+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:28:10.770437+0000","last_undegraded":"2023-10-14T01:28:10.770437+0000","last_fullsized":"2023-10-14T01:28:10.770437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'570357","reported_seq":1134678,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:12.931008+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:28:12.931008+0000","last_peered":"2023-10-14T01:28:12.931008+0000","last_clean":"2023-10-14T01:28:12.931008+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:28:12.931008+0000","last_undegraded":"2023-10-14T01:28:12.931008+0000","last_fullsized":"2023-10-14T01:28:12.931008+0000","mapping_epoch":15,"log_start":"99'560327","ondisk_log_start":"99'560327","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7574384640,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":566373,"num_read_kb":2233908,"num_write":565074,"num_write_kb":2260296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1565,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:05.649833+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:28:05.649833+0000","last_peered":"2023-10-14T01:28:05.649833+0000","last_clean":"2023-10-14T01:28:05.649833+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:28:05.649833+0000","last_undegraded":"2023-10-14T01:28:05.649833+0000","last_fullsized":"2023-10-14T01:28:05.649833+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'566673","reported_seq":1128717,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:14.204918+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:28:14.204918+0000","last_peered":"2023-10-14T01:28:14.204918+0000","last_clean":"2023-10-14T01:28:14.204918+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:28:14.204918+0000","last_undegraded":"2023-10-14T01:28:14.204918+0000","last_fullsized":"2023-10-14T01:28:14.204918+0000","mapping_epoch":15,"log_start":"99'556635","ondisk_log_start":"99'556635","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3221,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 548s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7553458195,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":563958,"num_read_kb":2223796,"num_write":561876,"num_write_kb":2247498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15339","reported_seq":20814,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:14.108748+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:28:14.108748+0000","last_peered":"2023-10-14T01:28:14.108748+0000","last_clean":"2023-10-14T01:28:14.108748+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:28:14.108748+0000","last_undegraded":"2023-10-14T01:28:14.108748+0000","last_fullsized":"2023-10-14T01:28:14.108748+0000","mapping_epoch":12,"log_start":"81'5300","ondisk_log_start":"81'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343110+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:47.343110+0000","last_peered":"2023-10-14T01:25:47.343110+0000","last_clean":"2023-10-14T01:25:47.343110+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:47.343110+0000","last_undegraded":"2023-10-14T01:25:47.343110+0000","last_fullsized":"2023-10-14T01:25:47.343110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30087729171,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2245685,"num_read_kb":8855660,"num_write":2241136,"num_write_kb":8964538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18356617216,"data_stored":21424644134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40234,"ondisk_log_size":40234,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48180,"ondisk_log_size":48180,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739374,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10572704,"kb_used_data":8961560,"kb_used_omap":1426,"kb_used_meta":1609709,"kb_avail":83799136,"statfs":{"total":96636764160,"available":85810315264,"internally_reserved":0,"allocated":9176637440,"data_stored":10710105897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":1648342135},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.19500000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":7.1040000000000001,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.375},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.16600000000000001},"max":{"1min":0.82599999999999996,"5min":3.835,"15min":3.835},"last":0.317}]},{"osd":1,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.61499999999999999,"15min":0.52200000000000002},"min":{"1min":0.23699999999999999,"5min":0.224,"15min":0.19800000000000001},"max":{"1min":7.1319999999999997,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66500000000000004,"15min":0.57199999999999995},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":7.1479999999999997,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.33900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739375,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5726604,"kb_used_data":4494660,"kb_used_omap":704,"kb_used_meta":1231231,"kb_avail":88645236,"statfs":{"total":96636764160,"available":90772721664,"internally_reserved":0,"allocated":4602531840,"data_stored":5369580200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1260781073},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,6,7,3,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:32 2023","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.41399999999999998,"15min":0.50800000000000001},"min":{"1min":0.17599999999999999,"5min":0.16900000000000001,"15min":0.158},"max":{"1min":2.0379999999999998,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.56799999999999995,"15min":0.55900000000000005},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.1190000000000002,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.35699999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:27:32 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.55800000000000005,"15min":0.51200000000000001},"min":{"1min":0.22900000000000001,"5min":0.187,"15min":0.186},"max":{"1min":0.495,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.245},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.64900000000000002,"15min":0.57499999999999996},"min":{"1min":0.26900000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51900000000000002,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.315}]}]},{"osd":0,"up_from":8,"seq":34359739377,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5671424,"kb_used_data":4472824,"kb_used_omap":753,"kb_used_meta":1197838,"kb_avail":88700416,"statfs":{"total":96636764160,"available":90829225984,"internally_reserved":0,"allocated":4580171776,"data_stored":5345670875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1226586869},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,1,3,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:27:16 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.35999999999999999,"15min":0.42999999999999999},"min":{"1min":0.19400000000000001,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":2.5259999999999998,"5min":2.7549999999999999,"15min":5.8600000000000003},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.22,"5min":0.187,"15min":0.187},"max":{"1min":2.625,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:27:16 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40500000000000003,"15min":0.63400000000000001},"min":{"1min":0.19700000000000001,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":1.8740000000000001,"5min":1.921,"15min":31.103000000000002},"last":0.215},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.54100000000000004},"min":{"1min":0.23799999999999999,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":0.55900000000000005,"5min":0.56799999999999995,"15min":30.234000000000002},"last":0.25800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4578537472,"data_stored":5344722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4601159680,"data_stored":5368987667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9176920064,"data_stored":10710933523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:28:18.553 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:18.553 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.2 2023-10-14T01:28:19.095 INFO:teuthology.orchestra.run.smithi105.stderr:instructing pg 3.2 on osd.1 to deep-scrub 2023-10-14T01:28:19.123 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:20.533 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:28:20.531+0000 7f120e762640 -1 reset not still connected to 0x5632b44076c0 2023-10-14T01:28:20.533 INFO:tasks.ceph.osd.0.smithi105.stderr:2023-10-14T01:28:20.531+0000 7f120e762640 -1 reset not still connected to 0x5632b4a13c70 2023-10-14T01:28:39.125 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:28:39.650 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:28:39.651 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:28:39.697 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2616,"stamp":"2023-10-14T01:28:38.556415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30126538572,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2277520,"num_read_kb":8912046,"num_write":2251276,"num_write_kb":9004115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88518,"ondisk_log_size":88518,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22247220,"kb_used_data":18006296,"kb_used_omap":2884,"kb_used_meta":4238011,"kb_avail":260868300,"statfs":{"total":289910292480,"available":267129139200,"internally_reserved":0,"allocated":18438447104,"data_stored":21504463020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":4339723645},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,8,12,17,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21237760,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5269,"num_read_kb":20992,"num_write":5190,"num_write_kb":20760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001975"},"pg_stats":[{"pgid":"2.7","version":"99'11634","reported_seq":15968,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.843640+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:28:32.843640+0000","last_peered":"2023-10-14T01:28:32.843640+0000","last_clean":"2023-10-14T01:28:32.843640+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:28:32.843640+0000","last_undegraded":"2023-10-14T01:28:32.843640+0000","last_fullsized":"2023-10-14T01:28:32.843640+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7832","reported_seq":11680,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.834240+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:28:32.834240+0000","last_peered":"2023-10-14T01:28:32.834240+0000","last_clean":"2023-10-14T01:28:32.834240+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:28:32.834240+0000","last_undegraded":"2023-10-14T01:28:32.834240+0000","last_fullsized":"2023-10-14T01:28:32.834240+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":189,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343196+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:25:47.343196+0000","last_peered":"2023-10-14T01:25:47.343196+0000","last_clean":"2023-10-14T01:25:47.343196+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:25:47.343196+0000","last_undegraded":"2023-10-14T01:25:47.343196+0000","last_fullsized":"2023-10-14T01:25:47.343196+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15465","reported_seq":21606,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.849621+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:28:32.849621+0000","last_peered":"2023-10-14T01:28:32.849621+0000","last_clean":"2023-10-14T01:28:32.849621+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:28:32.849621+0000","last_undegraded":"2023-10-14T01:28:32.849621+0000","last_fullsized":"2023-10-14T01:28:32.849621+0000","mapping_epoch":12,"log_start":"81'5400","ondisk_log_start":"81'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'560457","reported_seq":1114198,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:38.058180+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:28:38.058180+0000","last_peered":"2023-10-14T01:28:38.058180+0000","last_clean":"2023-10-14T01:28:38.058180+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:28:38.058180+0000","last_undegraded":"2023-10-14T01:28:38.058180+0000","last_fullsized":"2023-10-14T01:28:38.058180+0000","mapping_epoch":15,"log_start":"99'550434","ondisk_log_start":"99'550434","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7422488576,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":555744,"num_read_kb":2191572,"num_write":555284,"num_write_kb":2221136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11526","reported_seq":15751,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:37.948665+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:28:37.948665+0000","last_peered":"2023-10-14T01:28:37.948665+0000","last_clean":"2023-10-14T01:28:37.948665+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:28:37.948665+0000","last_undegraded":"2023-10-14T01:28:37.948665+0000","last_fullsized":"2023-10-14T01:28:37.948665+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'568807","reported_seq":1131076,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:34.325728+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:28:34.325728+0000","last_peered":"2023-10-14T01:28:34.325728+0000","last_clean":"2023-10-14T01:28:34.325728+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:28:34.325728+0000","last_undegraded":"2023-10-14T01:28:34.325728+0000","last_fullsized":"2023-10-14T01:28:34.325728+0000","mapping_epoch":15,"log_start":"99'558728","ondisk_log_start":"99'558728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7556431872,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":564301,"num_read_kb":2225072,"num_write":563538,"num_write_kb":2254152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4337,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:36.370374+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:28:36.370374+0000","last_peered":"2023-10-14T01:28:36.370374+0000","last_clean":"2023-10-14T01:28:36.370374+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:28:36.370374+0000","last_undegraded":"2023-10-14T01:28:36.370374+0000","last_fullsized":"2023-10-14T01:28:36.370374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'572935","reported_seq":1139825,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:38.067348+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:28:38.067348+0000","last_peered":"2023-10-14T01:28:38.067348+0000","last_clean":"2023-10-14T01:28:38.067348+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:28:38.067348+0000","last_undegraded":"2023-10-14T01:28:38.067348+0000","last_fullsized":"2023-10-14T01:28:38.067348+0000","mapping_epoch":15,"log_start":"99'562927","ondisk_log_start":"99'562927","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7584825344,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":568943,"num_read_kb":2244172,"num_write":567652,"num_write_kb":2270608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1570,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:31.249619+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:28:31.249619+0000","last_peered":"2023-10-14T01:28:31.249619+0000","last_clean":"2023-10-14T01:28:31.249619+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:28:31.249619+0000","last_undegraded":"2023-10-14T01:28:31.249619+0000","last_fullsized":"2023-10-14T01:28:31.249619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'568862","reported_seq":1133057,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:34.329276+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:28:34.329276+0000","last_peered":"2023-10-14T01:28:34.329276+0000","last_clean":"2023-10-14T01:28:34.329276+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:28:34.329276+0000","last_undegraded":"2023-10-14T01:28:34.329276+0000","last_fullsized":"2023-10-14T01:28:34.329276+0000","mapping_epoch":15,"log_start":"99'558835","ondisk_log_start":"99'558835","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3329,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 568s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7562199059,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":566095,"num_read_kb":2232316,"num_write":563992,"num_write_kb":2255962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15379","reported_seq":20870,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:32.854862+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:28:32.854862+0000","last_peered":"2023-10-14T01:28:32.854862+0000","last_clean":"2023-10-14T01:28:32.854862+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:28:32.854862+0000","last_undegraded":"2023-10-14T01:28:32.854862+0000","last_fullsized":"2023-10-14T01:28:32.854862+0000","mapping_epoch":12,"log_start":"81'5300","ondisk_log_start":"81'5300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343110+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:47.343110+0000","last_peered":"2023-10-14T01:25:47.343110+0000","last_clean":"2023-10-14T01:25:47.343110+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:47.343110+0000","last_undegraded":"2023-10-14T01:25:47.343110+0000","last_fullsized":"2023-10-14T01:25:47.343110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30125944851,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2255083,"num_read_kb":8893132,"num_write":2250466,"num_write_kb":9001858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18437226496,"data_stored":21505253414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40137,"ondisk_log_size":40137,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48282,"ondisk_log_size":48282,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739379,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10730316,"kb_used_data":9002564,"kb_used_omap":1426,"kb_used_meta":1726317,"kb_avail":83641524,"statfs":{"total":96636764160,"available":85648920576,"internally_reserved":0,"allocated":9218625536,"data_stored":10752093993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":1767748727},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.35899999999999999,"15min":0.36899999999999999},"min":{"1min":0.19500000000000001,"5min":0.183,"15min":0.17000000000000001},"max":{"1min":7.1040000000000001,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.433,"15min":0.375},"min":{"1min":0.20799999999999999,"5min":0.191,"15min":0.16600000000000001},"max":{"1min":0.82599999999999996,"5min":3.835,"15min":3.835},"last":0.27000000000000002}]},{"osd":1,"last update":"Sat Oct 14 01:27:50 2023","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.61499999999999999,"15min":0.52200000000000002},"min":{"1min":0.23699999999999999,"5min":0.224,"15min":0.19800000000000001},"max":{"1min":7.1319999999999997,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66500000000000004,"15min":0.57199999999999995},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":7.1479999999999997,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.30299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739379,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5783516,"kb_used_data":4511312,"kb_used_omap":704,"kb_used_meta":1271487,"kb_avail":88588324,"statfs":{"total":96636764160,"available":90714443776,"internally_reserved":0,"allocated":4619583488,"data_stored":5386631848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1302003217},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,5,3,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.438,"15min":0.51900000000000002},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.158},"max":{"1min":3.8029999999999999,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.843,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.218}]},{"osd":2,"last update":"Sat Oct 14 01:28:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.55900000000000005,"15min":0.51400000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":0.76300000000000001,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.65000000000000002,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.78500000000000003,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.45500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5733388,"kb_used_data":4492420,"kb_used_omap":753,"kb_used_meta":1240206,"kb_avail":88638452,"statfs":{"total":96636764160,"available":90765774848,"internally_reserved":0,"allocated":4600238080,"data_stored":5365737179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1269971701},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,7,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:19 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.42999999999999999},"min":{"1min":0.188,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.89500000000000002,"5min":2.7549999999999999,"15min":5.8600000000000003},"last":0.253},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.51600000000000001,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.53500000000000003,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.28599999999999998}]},{"osd":2,"last update":"Sat Oct 14 01:28:19 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.38600000000000001,"15min":0.61099999999999999},"min":{"1min":0.20899999999999999,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":0.44,"5min":1.921,"15min":31.103000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.33800000000000002,"15min":0.54400000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":30.234000000000002},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4599545856,"data_stored":5365731328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4618362880,"data_stored":5386190867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9219317760,"data_stored":10753331219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:28:39.699 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:28:39.699 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:28:59.501 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3935ea90 2023-10-14T01:28:59.502 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d39fcd860 2023-10-14T01:28:59.502 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a0c15f0 2023-10-14T01:28:59.503 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:28:59.503 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6000 2023-10-14T01:28:59.503 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:28:59.503 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:28:59.504 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:28:59.504 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6f70 2023-10-14T01:28:59.504 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7040 2023-10-14T01:28:59.505 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d7110 2023-10-14T01:28:59.505 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090410 2023-10-14T01:28:59.505 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:28:59.499+0000 7f5baaa31640 -1 reset not still connected to 0x562d3b090a90 2023-10-14T01:28:59.700 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:00.333 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:29:00.334 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:29:00.369 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2626,"stamp":"2023-10-14T01:28:58.562220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30160699212,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2285883,"num_read_kb":8945406,"num_write":2259651,"num_write_kb":9037615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88564,"ondisk_log_size":88564,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22474696,"kb_used_data":18075504,"kb_used_omap":2884,"kb_used_meta":4396283,"kb_avail":260640824,"statfs":{"total":289910292480,"available":266896203776,"internally_reserved":0,"allocated":18509316096,"data_stored":21575332012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":4501794173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,3,9,10,14,21,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21143552,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5165,"num_read_kb":20612,"num_write":5206,"num_write_kb":20824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004530"},"pg_stats":[{"pgid":"2.7","version":"99'11664","reported_seq":16009,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:54.104555+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:28:54.104555+0000","last_peered":"2023-10-14T01:28:54.104555+0000","last_clean":"2023-10-14T01:28:54.104555+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:28:54.104555+0000","last_undegraded":"2023-10-14T01:28:54.104555+0000","last_fullsized":"2023-10-14T01:28:54.104555+0000","mapping_epoch":12,"log_start":"57'1600","ondisk_log_start":"57'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7852","reported_seq":11708,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:54.081935+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:28:54.081935+0000","last_peered":"2023-10-14T01:28:54.081935+0000","last_clean":"2023-10-14T01:28:54.081935+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:28:54.081935+0000","last_undegraded":"2023-10-14T01:28:54.081935+0000","last_fullsized":"2023-10-14T01:28:54.081935+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":190,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.671760+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:26:33.671760+0000","last_peered":"2023-10-14T01:26:33.671760+0000","last_clean":"2023-10-14T01:26:33.671760+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:26:33.671760+0000","last_undegraded":"2023-10-14T01:26:33.671760+0000","last_fullsized":"2023-10-14T01:26:33.671760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15505","reported_seq":21660,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:54.109503+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:28:54.109503+0000","last_peered":"2023-10-14T01:28:54.109503+0000","last_clean":"2023-10-14T01:28:54.109503+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:28:54.109503+0000","last_undegraded":"2023-10-14T01:28:54.109503+0000","last_fullsized":"2023-10-14T01:28:54.109503+0000","mapping_epoch":12,"log_start":"81'5500","ondisk_log_start":"81'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'562574","reported_seq":1118379,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.176214+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:28:58.176214+0000","last_peered":"2023-10-14T01:28:58.176214+0000","last_clean":"2023-10-14T01:28:58.176214+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:28:58.176214+0000","last_undegraded":"2023-10-14T01:28:58.176214+0000","last_fullsized":"2023-10-14T01:28:58.176214+0000","mapping_epoch":15,"log_start":"99'552534","ondisk_log_start":"99'552534","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7431139328,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":557805,"num_read_kb":2199796,"num_write":557401,"num_write_kb":2229604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11566","reported_seq":15803,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:56.849635+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:28:56.849635+0000","last_peered":"2023-10-14T01:28:56.849635+0000","last_clean":"2023-10-14T01:28:56.849635+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:28:56.849635+0000","last_undegraded":"2023-10-14T01:28:56.849635+0000","last_fullsized":"2023-10-14T01:28:56.849635+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'570825","reported_seq":1135241,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:54.451670+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:28:54.451670+0000","last_peered":"2023-10-14T01:28:54.451670+0000","last_clean":"2023-10-14T01:28:54.451670+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:28:54.451670+0000","last_undegraded":"2023-10-14T01:28:54.451670+0000","last_fullsized":"2023-10-14T01:28:54.451670+0000","mapping_epoch":15,"log_start":"99'560728","ondisk_log_start":"99'560728","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7564763136,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":566449,"num_read_kb":2233636,"num_write":565556,"num_write_kb":2262224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4349,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:56.853669+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:28:56.853669+0000","last_peered":"2023-10-14T01:28:56.853669+0000","last_clean":"2023-10-14T01:28:56.853669+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:28:56.853669+0000","last_undegraded":"2023-10-14T01:28:56.853669+0000","last_fullsized":"2023-10-14T01:28:56.853669+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'575028","reported_seq":1144058,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:58.161774+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:28:58.161774+0000","last_peered":"2023-10-14T01:28:58.161774+0000","last_clean":"2023-10-14T01:28:58.161774+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:28:58.161774+0000","last_undegraded":"2023-10-14T01:28:58.161774+0000","last_fullsized":"2023-10-14T01:28:58.161774+0000","mapping_epoch":15,"log_start":"99'565027","ondisk_log_start":"99'565027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7593320448,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":571082,"num_read_kb":2252704,"num_write":569745,"num_write_kb":2278980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1574,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:51.734251+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:28:51.734251+0000","last_peered":"2023-10-14T01:28:51.734251+0000","last_clean":"2023-10-14T01:28:51.734251+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:28:51.734251+0000","last_undegraded":"2023-10-14T01:28:51.734251+0000","last_fullsized":"2023-10-14T01:28:51.734251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'571110","reported_seq":1137330,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:28:54.457739+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:28:54.457739+0000","last_peered":"2023-10-14T01:28:54.457739+0000","last_clean":"2023-10-14T01:28:54.457739+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:28:54.457739+0000","last_undegraded":"2023-10-14T01:28:54.457739+0000","last_fullsized":"2023-10-14T01:28:54.457739+0000","mapping_epoch":15,"log_start":"99'561035","ondisk_log_start":"99'561035","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3476,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 588s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7570882579,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":568110,"num_read_kb":2240356,"num_write":566139,"num_write_kb":2264550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15419","reported_seq":20926,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:28:54.106497+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:28:54.106497+0000","last_peered":"2023-10-14T01:28:54.106497+0000","last_clean":"2023-10-14T01:28:54.106497+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:28:54.106497+0000","last_undegraded":"2023-10-14T01:28:54.106497+0000","last_fullsized":"2023-10-14T01:28:54.106497+0000","mapping_epoch":12,"log_start":"81'5400","ondisk_log_start":"81'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":278,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:25:47.343110+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:25:47.343110+0000","last_peered":"2023-10-14T01:25:47.343110+0000","last_clean":"2023-10-14T01:25:47.343110+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:25:47.343110+0000","last_undegraded":"2023-10-14T01:25:47.343110+0000","last_fullsized":"2023-10-14T01:25:47.343110+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30160105491,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2263446,"num_read_kb":8926492,"num_write":2258841,"num_write_kb":9035358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18505900032,"data_stored":21573926950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40213,"ondisk_log_size":40213,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48252,"ondisk_log_size":48252,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739383,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10847160,"kb_used_data":9037488,"kb_used_omap":1426,"kb_used_meta":1808237,"kb_avail":83524680,"statfs":{"total":96636764160,"available":85529272320,"internally_reserved":0,"allocated":9254387712,"data_stored":10787856169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":1851634807},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.52400000000000002,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.437,"15min":0.38900000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.16600000000000001},"max":{"1min":1.6759999999999999,"5min":3.835,"15min":3.835},"last":0.74099999999999999}]},{"osd":1,"last update":"Sat Oct 14 01:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.59199999999999997,"15min":0.52300000000000002},"min":{"1min":0.23200000000000001,"5min":0.224,"15min":0.20399999999999999},"max":{"1min":0.55500000000000005,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.63900000000000001,"15min":0.57199999999999995},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.39700000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739383,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5836240,"kb_used_data":4527944,"kb_used_omap":704,"kb_used_meta":1307583,"kb_avail":88535600,"statfs":{"total":96636764160,"available":90660454400,"internally_reserved":0,"allocated":4636614656,"data_stored":5403663016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1338965521},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,4,4,5,17,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.438,"15min":0.51900000000000002},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.158},"max":{"1min":3.8029999999999999,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.843,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.219}]},{"osd":2,"last update":"Sat Oct 14 01:28:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.55900000000000005,"15min":0.51400000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":0.76300000000000001,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.65000000000000002,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.78500000000000003,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5791296,"kb_used_data":4510072,"kb_used_omap":753,"kb_used_meta":1280462,"kb_avail":88580544,"statfs":{"total":96636764160,"available":90706477056,"internally_reserved":0,"allocated":4618313728,"data_stored":5383812827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1311193845},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:19 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.42999999999999999},"min":{"1min":0.188,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.89500000000000002,"5min":2.7549999999999999,"15min":5.8600000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.51600000000000001,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.53500000000000003,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:28:19 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.38600000000000001,"15min":0.61099999999999999},"min":{"1min":0.20899999999999999,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":0.44,"5min":1.921,"15min":31.103000000000002},"last":0.25},{"interface":"front","average":{"1min":0.375,"5min":0.33800000000000002,"15min":0.54400000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":30.234000000000002},"last":0.38100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4616790016,"data_stored":5382975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4635418624,"data_stored":5403246611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9253691392,"data_stored":10787704851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:29:00.371 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:00.371 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:20.373 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:20.962 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:29:20.963 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:29:21.000 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2637,"stamp":"2023-10-14T01:29:20.565945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30199320396,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2295266,"num_read_kb":8982862,"num_write":2269157,"num_write_kb":9075639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88681,"ondisk_log_size":88681,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":22757460,"kb_used_data":18148988,"kb_used_omap":2884,"kb_used_meta":4605563,"kb_avail":260358060,"statfs":{"total":289910292480,"available":266606653440,"internally_reserved":0,"allocated":18584563712,"data_stored":21650579628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":4716096893},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,6,8,14,10,22,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21770240,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5218,"num_read_kb":20840,"num_write":5345,"num_write_kb":21380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002065"},"pg_stats":[{"pgid":"2.7","version":"99'11709","reported_seq":16069,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.468837+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:29:18.468837+0000","last_peered":"2023-10-14T01:29:18.468837+0000","last_clean":"2023-10-14T01:29:18.468837+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:29:18.468837+0000","last_undegraded":"2023-10-14T01:29:18.468837+0000","last_fullsized":"2023-10-14T01:29:18.468837+0000","mapping_epoch":12,"log_start":"63'1700","ondisk_log_start":"63'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7882","reported_seq":11748,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.447264+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:29:18.447264+0000","last_peered":"2023-10-14T01:29:18.447264+0000","last_clean":"2023-10-14T01:29:18.447264+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:29:18.447264+0000","last_undegraded":"2023-10-14T01:29:18.447264+0000","last_fullsized":"2023-10-14T01:29:18.447264+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7882,"log_dups_size":0,"ondisk_log_size":7882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":190,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.671760+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:26:33.671760+0000","last_peered":"2023-10-14T01:26:33.671760+0000","last_clean":"2023-10-14T01:26:33.671760+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:26:33.671760+0000","last_undegraded":"2023-10-14T01:26:33.671760+0000","last_fullsized":"2023-10-14T01:26:33.671760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15565","reported_seq":21740,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.468830+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:29:18.468830+0000","last_peered":"2023-10-14T01:29:18.468830+0000","last_clean":"2023-10-14T01:29:18.468830+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:29:18.468830+0000","last_undegraded":"2023-10-14T01:29:18.468830+0000","last_fullsized":"2023-10-14T01:29:18.468830+0000","mapping_epoch":12,"log_start":"81'5500","ondisk_log_start":"81'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'564659","reported_seq":1122513,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.200831+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:29:18.200831+0000","last_peered":"2023-10-14T01:29:18.200831+0000","last_clean":"2023-10-14T01:29:18.200831+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:29:18.200831+0000","last_undegraded":"2023-10-14T01:29:18.200831+0000","last_fullsized":"2023-10-14T01:29:18.200831+0000","mapping_epoch":15,"log_start":"99'554634","ondisk_log_start":"99'554634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7439552512,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":559854,"num_read_kb":2207964,"num_write":559486,"num_write_kb":2237944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11584","reported_seq":15833,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:17.329826+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:29:17.329826+0000","last_peered":"2023-10-14T01:29:17.329826+0000","last_clean":"2023-10-14T01:29:17.329826+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:29:17.329826+0000","last_undegraded":"2023-10-14T01:29:17.329826+0000","last_fullsized":"2023-10-14T01:29:17.329826+0000","mapping_epoch":12,"log_start":"57'1500","ondisk_log_start":"57'1500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'573497","reported_seq":1140507,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:19.661236+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:29:19.661236+0000","last_peered":"2023-10-14T01:29:19.661236+0000","last_clean":"2023-10-14T01:29:19.661236+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:29:19.661236+0000","last_undegraded":"2023-10-14T01:29:19.661236+0000","last_fullsized":"2023-10-14T01:29:19.661236+0000","mapping_epoch":15,"log_start":"99'563428","ondisk_log_start":"99'563428","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7575576576,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":569042,"num_read_kb":2243980,"num_write":568228,"num_write_kb":2272912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4361,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:17.334363+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:29:17.334363+0000","last_peered":"2023-10-14T01:29:17.334363+0000","last_clean":"2023-10-14T01:29:17.334363+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:29:17.334363+0000","last_undegraded":"2023-10-14T01:29:17.334363+0000","last_fullsized":"2023-10-14T01:29:17.334363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'577126","reported_seq":1148223,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.183850+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:29:18.183850+0000","last_peered":"2023-10-14T01:29:18.183850+0000","last_clean":"2023-10-14T01:29:18.183850+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:29:18.183850+0000","last_undegraded":"2023-10-14T01:29:18.183850+0000","last_fullsized":"2023-10-14T01:29:18.183850+0000","mapping_epoch":15,"log_start":"99'567027","ondisk_log_start":"99'567027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7601848320,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":573151,"num_read_kb":2260972,"num_write":571843,"num_write_kb":2287372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1579,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:17.329614+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:29:17.329614+0000","last_peered":"2023-10-14T01:29:17.329614+0000","last_clean":"2023-10-14T01:29:17.329614+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:29:17.329614+0000","last_undegraded":"2023-10-14T01:29:17.329614+0000","last_fullsized":"2023-10-14T01:29:17.329614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'573859","reported_seq":1142766,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:19.666950+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:29:19.666950+0000","last_peered":"2023-10-14T01:29:19.666950+0000","last_clean":"2023-10-14T01:29:19.666950+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:29:19.666950+0000","last_undegraded":"2023-10-14T01:29:19.666950+0000","last_fullsized":"2023-10-14T01:29:19.666950+0000","mapping_epoch":15,"log_start":"99'563835","ondisk_log_start":"99'563835","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3620,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 613s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7581749267,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":570782,"num_read_kb":2251032,"num_write":568790,"num_write_kb":2275154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15479","reported_seq":21006,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:18.480027+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:29:18.480027+0000","last_peered":"2023-10-14T01:29:18.480027+0000","last_clean":"2023-10-14T01:29:18.480027+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:29:18.480027+0000","last_undegraded":"2023-10-14T01:29:18.480027+0000","last_fullsized":"2023-10-14T01:29:18.480027+0000","mapping_epoch":12,"log_start":"81'5400","ondisk_log_start":"81'5400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":279,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.914771+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:27:13.914771+0000","last_peered":"2023-10-14T01:27:13.914771+0000","last_clean":"2023-10-14T01:27:13.914771+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:27:13.914771+0000","last_undegraded":"2023-10-14T01:27:13.914771+0000","last_fullsized":"2023-10-14T01:27:13.914771+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30198726675,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2272829,"num_read_kb":8963948,"num_write":2268347,"num_write_kb":9073382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18581581824,"data_stored":21649608742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40217,"ondisk_log_size":40217,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48365,"ondisk_log_size":48365,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739388,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":10973064,"kb_used_data":9073152,"kb_used_omap":1426,"kb_used_meta":1898477,"kb_avail":83398776,"statfs":{"total":96636764160,"available":85400346624,"internally_reserved":0,"allocated":9290907648,"data_stored":10824376105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":1944040567},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.52400000000000002,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.316},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.437,"15min":0.38900000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.16600000000000001},"max":{"1min":1.6759999999999999,"5min":3.835,"15min":3.835},"last":0.46999999999999997}]},{"osd":1,"last update":"Sat Oct 14 01:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.59199999999999997,"15min":0.52300000000000002},"min":{"1min":0.23200000000000001,"5min":0.224,"15min":0.20399999999999999},"max":{"1min":0.55500000000000005,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.503},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.63900000000000001,"15min":0.57199999999999995},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.34999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739388,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5940268,"kb_used_data":4549028,"kb_used_omap":704,"kb_used_meta":1390527,"kb_avail":88431572,"statfs":{"total":96636764160,"available":90553929728,"internally_reserved":0,"allocated":4658204672,"data_stored":5425253032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1423900177},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,1,8,3,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.438,"15min":0.51900000000000002},"min":{"1min":0.17699999999999999,"5min":0.17599999999999999,"15min":0.158},"max":{"1min":3.8029999999999999,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.843,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:28:34 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.55900000000000005,"15min":0.51400000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":0.76300000000000001,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.65000000000000002,"15min":0.57799999999999996},"min":{"1min":0.28299999999999997,"5min":0.251,"15min":0.23899999999999999},"max":{"1min":0.78500000000000003,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5844128,"kb_used_data":4526808,"kb_used_omap":753,"kb_used_meta":1316558,"kb_avail":88527712,"statfs":{"total":96636764160,"available":90652377088,"internally_reserved":0,"allocated":4635451392,"data_stored":5400950491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1348156149},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,7,6,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:28:19 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.375,"15min":0.42999999999999999},"min":{"1min":0.188,"5min":0.184,"15min":0.17399999999999999},"max":{"1min":0.89500000000000002,"5min":2.7549999999999999,"15min":5.8600000000000003},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.51600000000000001,"15min":0.54700000000000004},"min":{"1min":0.20300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.53500000000000003,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Oct 14 01:28:19 2023","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.38600000000000001,"15min":0.61099999999999999},"min":{"1min":0.20899999999999999,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":0.44,"5min":1.921,"15min":31.103000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.33800000000000002,"15min":0.54400000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":30.234000000000002},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4633931776,"data_stored":5400117248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4657221632,"data_stored":5425049619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9290428416,"data_stored":10824441875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:29:21.002 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:21.003 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:29:41.005 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:29:41.493 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:29:41.494 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:29:41.542 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2647,"stamp":"2023-10-14T01:29:40.569159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30232670028,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2303162,"num_read_kb":9014382,"num_write":2277374,"num_write_kb":9108507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88664,"ondisk_log_size":88664,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23062676,"kb_used_data":18214008,"kb_used_omap":2884,"kb_used_meta":4845755,"kb_avail":260052844,"statfs":{"total":289910292480,"available":266294112256,"internally_reserved":0,"allocated":18651144192,"data_stored":21717160108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":4962053501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,3,8,21,19],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21368832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5012,"num_read_kb":20000,"num_write":5265,"num_write_kb":21060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001963"},"pg_stats":[{"pgid":"2.7","version":"99'11760","reported_seq":16132,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.845725+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:29:38.845725+0000","last_peered":"2023-10-14T01:29:38.845725+0000","last_clean":"2023-10-14T01:29:38.845725+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:29:38.845725+0000","last_undegraded":"2023-10-14T01:29:38.845725+0000","last_fullsized":"2023-10-14T01:29:38.845725+0000","mapping_epoch":12,"log_start":"63'1700","ondisk_log_start":"63'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7916","reported_seq":11791,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.852808+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:29:38.852808+0000","last_peered":"2023-10-14T01:29:38.852808+0000","last_clean":"2023-10-14T01:29:38.852808+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:29:38.852808+0000","last_undegraded":"2023-10-14T01:29:38.852808+0000","last_fullsized":"2023-10-14T01:29:38.852808+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7916,"log_dups_size":0,"ondisk_log_size":7916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":190,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.671760+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:26:33.671760+0000","last_peered":"2023-10-14T01:26:33.671760+0000","last_clean":"2023-10-14T01:26:33.671760+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:26:33.671760+0000","last_undegraded":"2023-10-14T01:26:33.671760+0000","last_fullsized":"2023-10-14T01:26:33.671760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15633","reported_seq":21824,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.855500+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:29:38.855500+0000","last_peered":"2023-10-14T01:29:38.855500+0000","last_clean":"2023-10-14T01:29:38.855500+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:29:38.855500+0000","last_undegraded":"2023-10-14T01:29:38.855500+0000","last_fullsized":"2023-10-14T01:29:38.855500+0000","mapping_epoch":12,"log_start":"81'5600","ondisk_log_start":"81'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'566679","reported_seq":1126391,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.319532+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:29:38.319532+0000","last_peered":"2023-10-14T01:29:38.319532+0000","last_clean":"2023-10-14T01:29:38.319532+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:29:38.319532+0000","last_undegraded":"2023-10-14T01:29:38.319532+0000","last_fullsized":"2023-10-14T01:29:38.319532+0000","mapping_epoch":15,"log_start":"99'556634","ondisk_log_start":"99'556634","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7447756800,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":561713,"num_read_kb":2215392,"num_write":561506,"num_write_kb":2246024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11617","reported_seq":15878,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:37.809672+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:29:37.809672+0000","last_peered":"2023-10-14T01:29:37.809672+0000","last_clean":"2023-10-14T01:29:37.809672+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:29:37.809672+0000","last_undegraded":"2023-10-14T01:29:37.809672+0000","last_fullsized":"2023-10-14T01:29:37.809672+0000","mapping_epoch":12,"log_start":"61'1600","ondisk_log_start":"61'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'575580","reported_seq":1144547,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:39.761297+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:29:39.761297+0000","last_peered":"2023-10-14T01:29:39.761297+0000","last_clean":"2023-10-14T01:29:39.761297+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:29:39.761297+0000","last_undegraded":"2023-10-14T01:29:39.761297+0000","last_fullsized":"2023-10-14T01:29:39.761297+0000","mapping_epoch":15,"log_start":"99'565528","ondisk_log_start":"99'565528","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7584047104,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":570999,"num_read_kb":2251784,"num_write":570311,"num_write_kb":2281244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4373,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:37.833953+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:29:37.833953+0000","last_peered":"2023-10-14T01:29:37.833953+0000","last_clean":"2023-10-14T01:29:37.833953+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:29:37.833953+0000","last_undegraded":"2023-10-14T01:29:37.833953+0000","last_fullsized":"2023-10-14T01:29:37.833953+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'579121","reported_seq":1152228,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.319513+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:29:38.319513+0000","last_peered":"2023-10-14T01:29:38.319513+0000","last_clean":"2023-10-14T01:29:38.319513+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:29:38.319513+0000","last_undegraded":"2023-10-14T01:29:38.319513+0000","last_fullsized":"2023-10-14T01:29:38.319513+0000","mapping_epoch":15,"log_start":"99'569027","ondisk_log_start":"99'569027","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7609917440,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":575161,"num_read_kb":2269000,"num_write":573838,"num_write_kb":2295352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1583,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:32.689663+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:29:32.689663+0000","last_peered":"2023-10-14T01:29:32.689663+0000","last_clean":"2023-10-14T01:29:32.689663+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:29:32.689663+0000","last_undegraded":"2023-10-14T01:29:32.689663+0000","last_fullsized":"2023-10-14T01:29:32.689663+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'576090","reported_seq":1147081,"reported_epoch":99,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-14T01:29:39.763548+0000","last_change":"2023-10-14T01:19:06.076896+0000","last_active":"2023-10-14T01:29:39.763548+0000","last_peered":"2023-10-14T01:29:39.763548+0000","last_clean":"2023-10-14T01:29:39.763548+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:29:39.763548+0000","last_undegraded":"2023-10-14T01:29:39.763548+0000","last_fullsized":"2023-10-14T01:29:39.763548+0000","mapping_epoch":15,"log_start":"99'566035","ondisk_log_start":"99'566035","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T00:04:52.902778+0000","last_clean_scrub_stamp":"2023-10-14T00:04:52.902778+0000","objects_scrubbed":3782,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 633s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7590354963,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":572852,"num_read_kb":2259292,"num_write":570909,"num_write_kb":2283630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15547","reported_seq":21090,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:38.894118+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:29:38.894118+0000","last_peered":"2023-10-14T01:29:38.894118+0000","last_clean":"2023-10-14T01:29:38.894118+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:29:38.894118+0000","last_undegraded":"2023-10-14T01:29:38.894118+0000","last_fullsized":"2023-10-14T01:29:38.894118+0000","mapping_epoch":12,"log_start":"81'5500","ondisk_log_start":"81'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":279,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.914771+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:27:13.914771+0000","last_peered":"2023-10-14T01:27:13.914771+0000","last_clean":"2023-10-14T01:27:13.914771+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:27:13.914771+0000","last_undegraded":"2023-10-14T01:27:13.914771+0000","last_fullsized":"2023-10-14T01:27:13.914771+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30232076307,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2280725,"num_read_kb":8995468,"num_write":2276564,"num_write_kb":9106250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18648817664,"data_stored":21716844582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40246,"ondisk_log_size":40246,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48319,"ondisk_log_size":48319,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739392,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11119448,"kb_used_data":9105616,"kb_used_omap":1426,"kb_used_meta":2012397,"kb_avail":83252392,"statfs":{"total":96636764160,"available":85250449408,"internally_reserved":0,"allocated":9324150784,"data_stored":10857619241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":2060694647},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.36299999999999999,"15min":0.36599999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.52400000000000002,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.437,"15min":0.38900000000000001},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.16600000000000001},"max":{"1min":1.6759999999999999,"5min":3.835,"15min":3.835},"last":0.45100000000000001}]},{"osd":1,"last update":"Sat Oct 14 01:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.59199999999999997,"15min":0.52300000000000002},"min":{"1min":0.23200000000000001,"5min":0.224,"15min":0.20399999999999999},"max":{"1min":0.55500000000000005,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.63900000000000001,"15min":0.57199999999999995},"min":{"1min":0.30299999999999999,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.52600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739392,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6004928,"kb_used_data":4565752,"kb_used_omap":704,"kb_used_meta":1438463,"kb_avail":88366912,"statfs":{"total":96636764160,"available":90487717888,"internally_reserved":0,"allocated":4675330048,"data_stored":5442378408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1472986641},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,11,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.505,"15min":0.52500000000000002},"min":{"1min":0.18099999999999999,"5min":0.17599999999999999,"15min":0.158},"max":{"1min":4.3479999999999999,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.27900000000000003,"5min":0.57299999999999995,"15min":0.54700000000000004},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.42499999999999999,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Oct 14 01:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.20799999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":0.40100000000000002,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.64600000000000002,"15min":0.56000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.23899999999999999},"max":{"1min":0.42599999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739395,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5938300,"kb_used_data":4542640,"kb_used_omap":753,"kb_used_meta":1394894,"kb_avail":88433540,"statfs":{"total":96636764160,"available":90555944960,"internally_reserved":0,"allocated":4651663360,"data_stored":5417162459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1428372213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,4,7,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.35799999999999998,"15min":0.41999999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17399999999999999},"max":{"1min":1.617,"5min":2.5259999999999998,"15min":5.8600000000000003},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.219,"5min":0.187,"15min":0.187},"max":{"1min":3.7360000000000002,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":1.395}]},{"osd":2,"last update":"Sat Oct 14 01:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.442,"15min":0.625},"min":{"1min":0.21099999999999999,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":5.5380000000000003,"5min":5.5380000000000003,"15min":31.103000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.376,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":3.5289999999999999,"5min":3.5289999999999999,"15min":30.234000000000002},"last":0.23699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4650385408,"data_stored":5416570880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4674437120,"data_stored":5442265107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9323995136,"data_stored":10858008595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:29:41.543 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-14T00:04:52.902778+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=0, tm_min=4, tm_sec=52, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=1, tm_min=10, tm_sec=12, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T01:29:41.544 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T01:30:01.545 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T01:30:02.110 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-10-14T01:30:02.111 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2023-10-14T01:30:02.147 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":2657,"stamp":"2023-10-14T01:30:00.572382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30267625292,"num_objects":15466,"num_object_clones":10240,"num_object_copies":30932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15466,"num_whiteouts":0,"num_read":2311718,"num_read_kb":9048526,"num_write":2285958,"num_write_kb":9142843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88706,"ondisk_log_size":88706,"up":26,"acting":26,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":23284668,"kb_used_data":18282084,"kb_used_omap":2884,"kb_used_meta":4999675,"kb_avail":259830852,"statfs":{"total":289910292480,"available":266066792448,"internally_reserved":0,"allocated":18720854016,"data_stored":21786869932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2953859,"internal_metadata":5119667581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,6,3,7,11,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21995520,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5464,"num_read_kb":21808,"num_write":5402,"num_write_kb":21608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001848"},"pg_stats":[{"pgid":"2.7","version":"99'11799","reported_seq":16184,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:59.693252+0000","last_change":"2023-10-14T01:10:25.978764+0000","last_active":"2023-10-14T01:29:59.693252+0000","last_peered":"2023-10-14T01:29:59.693252+0000","last_clean":"2023-10-14T01:29:59.693252+0000","last_became_active":"2023-10-14T00:04:47.892384+0000","last_became_peered":"2023-10-14T00:04:47.892384+0000","last_unstale":"2023-10-14T01:29:59.693252+0000","last_undegraded":"2023-10-14T01:29:59.693252+0000","last_fullsized":"2023-10-14T01:29:59.693252+0000","mapping_epoch":12,"log_start":"63'1700","ondisk_log_start":"63'1700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9441","last_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_deep_scrub":"85'9441","last_deep_scrub_stamp":"2023-10-14T01:10:25.978102+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.978102+0000","objects_scrubbed":11,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T02:21:38.650161+0000","scrub_duration":0.13629843699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3070,"num_read_kb":2548,"num_write":101,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"99'7942","reported_seq":11825,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:59.663548+0000","last_change":"2023-10-14T01:10:22.965372+0000","last_active":"2023-10-14T01:29:59.663548+0000","last_peered":"2023-10-14T01:29:59.663548+0000","last_clean":"2023-10-14T01:29:59.663548+0000","last_became_active":"2023-10-14T00:04:47.892212+0000","last_became_peered":"2023-10-14T00:04:47.892212+0000","last_unstale":"2023-10-14T01:29:59.663548+0000","last_undegraded":"2023-10-14T01:29:59.663548+0000","last_fullsized":"2023-10-14T01:29:59.663548+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'6360","last_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_deep_scrub":"85'6360","last_deep_scrub_stamp":"2023-10-14T01:10:22.965153+0000","last_clean_scrub_stamp":"2023-10-14T01:10:22.965153+0000","objects_scrubbed":15,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T07:46:03.616177+0000","scrub_duration":0.11543084300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":575,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3701,"num_read_kb":3342,"num_write":107,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2882,"num_omap_keys":67,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"85'24","reported_seq":190,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:26:33.671760+0000","last_change":"2023-10-14T01:10:21.898506+0000","last_active":"2023-10-14T01:26:33.671760+0000","last_peered":"2023-10-14T01:26:33.671760+0000","last_clean":"2023-10-14T01:26:33.671760+0000","last_became_active":"2023-10-14T00:04:47.893001+0000","last_became_peered":"2023-10-14T00:04:47.893001+0000","last_unstale":"2023-10-14T01:26:33.671760+0000","last_undegraded":"2023-10-14T01:26:33.671760+0000","last_fullsized":"2023-10-14T01:26:33.671760+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'24","last_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_deep_scrub":"85'24","last_deep_scrub_stamp":"2023-10-14T01:10:21.898266+0000","last_clean_scrub_stamp":"2023-10-14T01:10:21.898266+0000","objects_scrubbed":12,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T05:04:54.701664+0000","scrub_duration":0.021352332000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":39,"num_read_kb":12,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"99'15685","reported_seq":21892,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:59.712951+0000","last_change":"2023-10-14T01:10:20.122224+0000","last_active":"2023-10-14T01:29:59.712951+0000","last_peered":"2023-10-14T01:29:59.712951+0000","last_clean":"2023-10-14T01:29:59.712951+0000","last_became_active":"2023-10-14T00:04:47.892163+0000","last_became_peered":"2023-10-14T00:04:47.892163+0000","last_unstale":"2023-10-14T01:29:59.712951+0000","last_undegraded":"2023-10-14T01:29:59.712951+0000","last_fullsized":"2023-10-14T01:29:59.712951+0000","mapping_epoch":12,"log_start":"81'5600","ondisk_log_start":"81'5600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12551","last_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_deep_scrub":"85'12551","last_deep_scrub_stamp":"2023-10-14T01:10:20.121924+0000","last_clean_scrub_stamp":"2023-10-14T01:10:20.121924+0000","objects_scrubbed":20,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:59:30.689095+0000","scrub_duration":0.19077482100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4580,"num_read_kb":3852,"num_write":145,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3471,"num_omap_keys":88,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"99'568841","reported_seq":1130601,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.560346+0000","last_change":"2023-10-14T01:19:03.598712+0000","last_active":"2023-10-14T01:29:58.560346+0000","last_peered":"2023-10-14T01:29:58.560346+0000","last_clean":"2023-10-14T01:29:58.560346+0000","last_became_active":"2023-10-14T00:04:54.071087+0000","last_became_peered":"2023-10-14T00:04:54.071087+0000","last_unstale":"2023-10-14T01:29:58.560346+0000","last_undegraded":"2023-10-14T01:29:58.560346+0000","last_fullsized":"2023-10-14T01:29:58.560346+0000","mapping_epoch":15,"log_start":"99'558834","ondisk_log_start":"99'558834","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'502581","last_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_deep_scrub":"93'502581","last_deep_scrub_stamp":"2023-10-14T01:19:03.598234+0000","last_clean_scrub_stamp":"2023-10-14T01:19:03.598234+0000","objects_scrubbed":3783,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":521,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T04:04:28.303047+0000","scrub_duration":520.38160966099997,"objects_trimmed":0,"snaptrim_duration":0.0010931649999999999,"stat_sum":{"num_bytes":7456595968,"num_objects":3783,"num_object_clones":2522,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":563761,"num_read_kb":2223564,"num_write":563668,"num_write_kb":2254672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"99'11653","reported_seq":15926,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.289731+0000","last_change":"2023-10-14T01:10:18.577350+0000","last_active":"2023-10-14T01:29:58.289731+0000","last_peered":"2023-10-14T01:29:58.289731+0000","last_clean":"2023-10-14T01:29:58.289731+0000","last_became_active":"2023-10-14T00:04:47.889721+0000","last_became_peered":"2023-10-14T00:04:47.889721+0000","last_unstale":"2023-10-14T01:29:58.289731+0000","last_undegraded":"2023-10-14T01:29:58.289731+0000","last_fullsized":"2023-10-14T01:29:58.289731+0000","mapping_epoch":12,"log_start":"61'1600","ondisk_log_start":"61'1600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9291","last_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_deep_scrub":"85'9291","last_deep_scrub_stamp":"2023-10-14T01:10:18.577134+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.577134+0000","objects_scrubbed":14,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:21:02.487344+0000","scrub_duration":0.40239657099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3019,"num_read_kb":2516,"num_write":101,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3565,"num_omap_keys":56,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"99'577668","reported_seq":1148799,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:59.824712+0000","last_change":"2023-10-14T01:19:54.642443+0000","last_active":"2023-10-14T01:29:59.824712+0000","last_peered":"2023-10-14T01:29:59.824712+0000","last_clean":"2023-10-14T01:29:59.824712+0000","last_became_active":"2023-10-14T00:04:54.069172+0000","last_became_peered":"2023-10-14T00:04:54.069172+0000","last_unstale":"2023-10-14T01:29:59.824712+0000","last_undegraded":"2023-10-14T01:29:59.824712+0000","last_fullsized":"2023-10-14T01:29:59.824712+0000","mapping_epoch":15,"log_start":"99'567628","ondisk_log_start":"99'567628","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'515222","last_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_deep_scrub":"94'515222","last_deep_scrub_stamp":"2023-10-14T01:19:54.642260+0000","last_clean_scrub_stamp":"2023-10-14T01:19:54.642260+0000","objects_scrubbed":3855,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":567,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T06:16:14.060960+0000","scrub_duration":566.80635646999997,"objects_trimmed":0,"snaptrim_duration":0.00091703599999999998,"stat_sum":{"num_bytes":7592579072,"num_objects":3855,"num_object_clones":2570,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3855,"num_whiteouts":0,"num_read":573166,"num_read_kb":2260432,"num_write":572399,"num_write_kb":2289596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"99'152","reported_seq":4385,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.289911+0000","last_change":"2023-10-14T01:10:17.652928+0000","last_active":"2023-10-14T01:29:58.289911+0000","last_peered":"2023-10-14T01:29:58.289911+0000","last_clean":"2023-10-14T01:29:58.289911+0000","last_became_active":"2023-10-14T00:04:47.889680+0000","last_became_peered":"2023-10-14T00:04:47.889680+0000","last_unstale":"2023-10-14T01:29:58.289911+0000","last_undegraded":"2023-10-14T01:29:58.289911+0000","last_fullsized":"2023-10-14T01:29:58.289911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'146","last_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_deep_scrub":"85'146","last_deep_scrub_stamp":"2023-10-14T01:10:17.652736+0000","last_clean_scrub_stamp":"2023-10-14T01:10:17.652736+0000","objects_scrubbed":7,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T01:26:44.540544+0000","scrub_duration":0.053273893000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2977,"num_read_kb":2472,"num_write":77,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2205,"num_omap_keys":36,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"99'581249","reported_seq":1156515,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:58.540196+0000","last_change":"2023-10-14T01:19:36.593840+0000","last_active":"2023-10-14T01:29:58.540196+0000","last_peered":"2023-10-14T01:29:58.540196+0000","last_clean":"2023-10-14T01:29:58.540196+0000","last_became_active":"2023-10-14T00:04:54.070956+0000","last_became_peered":"2023-10-14T00:04:54.070956+0000","last_unstale":"2023-10-14T01:29:58.540196+0000","last_undegraded":"2023-10-14T01:29:58.540196+0000","last_fullsized":"2023-10-14T01:29:58.540196+0000","mapping_epoch":15,"log_start":"99'571227","ondisk_log_start":"99'571227","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'516932","last_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_deep_scrub":"93'516932","last_deep_scrub_stamp":"2023-10-14T01:19:36.593659+0000","last_clean_scrub_stamp":"2023-10-14T01:19:36.593659+0000","objects_scrubbed":3864,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":558,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:37:40.807097+0000","scrub_duration":557.41941806600005,"objects_trimmed":0,"snaptrim_duration":0.0012202669999999999,"stat_sum":{"num_bytes":7618564096,"num_objects":3864,"num_object_clones":2576,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":577324,"num_read_kb":2277632,"num_write":575966,"num_write_kb":2303864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"99'70","reported_seq":1587,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:53.169862+0000","last_change":"2023-10-14T01:10:18.544349+0000","last_active":"2023-10-14T01:29:53.169862+0000","last_peered":"2023-10-14T01:29:53.169862+0000","last_clean":"2023-10-14T01:29:53.169862+0000","last_became_active":"2023-10-14T00:04:47.890087+0000","last_became_peered":"2023-10-14T00:04:47.890087+0000","last_unstale":"2023-10-14T01:29:53.169862+0000","last_undegraded":"2023-10-14T01:29:53.169862+0000","last_fullsized":"2023-10-14T01:29:53.169862+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'68","last_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_deep_scrub":"85'68","last_deep_scrub_stamp":"2023-10-14T01:10:18.544154+0000","last_clean_scrub_stamp":"2023-10-14T01:10:18.544154+0000","objects_scrubbed":13,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T09:23:25.650896+0000","scrub_duration":0.028786798999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1018,"num_read_kb":827,"num_write":35,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":735,"num_omap_keys":12,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"99'578349","reported_seq":1151531,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:59.840281+0000","last_change":"2023-10-14T01:29:52.122382+0000","last_active":"2023-10-14T01:29:59.840281+0000","last_peered":"2023-10-14T01:29:59.840281+0000","last_clean":"2023-10-14T01:29:59.840281+0000","last_became_active":"2023-10-14T00:04:54.070581+0000","last_became_peered":"2023-10-14T00:04:54.070581+0000","last_unstale":"2023-10-14T01:29:59.840281+0000","last_undegraded":"2023-10-14T01:29:59.840281+0000","last_fullsized":"2023-10-14T01:29:59.840281+0000","mapping_epoch":15,"log_start":"99'568335","ondisk_log_start":"99'568335","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'577476","last_scrub_stamp":"2023-10-14T01:29:52.122163+0000","last_deep_scrub":"99'577476","last_deep_scrub_stamp":"2023-10-14T01:29:52.122163+0000","last_clean_scrub_stamp":"2023-10-14T01:29:52.122163+0000","objects_scrubbed":3859,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":647,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T10:45:40.104946+0000","scrub_duration":646.04443153800003,"objects_trimmed":0,"snaptrim_duration":0.0024264730000000002,"stat_sum":{"num_bytes":7599292435,"num_objects":3859,"num_object_clones":2572,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":575030,"num_read_kb":2267984,"num_write":573115,"num_write_kb":2292454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"99'15599","reported_seq":21158,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:29:59.745287+0000","last_change":"2023-10-14T01:10:25.235024+0000","last_active":"2023-10-14T01:29:59.745287+0000","last_peered":"2023-10-14T01:29:59.745287+0000","last_clean":"2023-10-14T01:29:59.745287+0000","last_became_active":"2023-10-14T00:04:47.887621+0000","last_became_peered":"2023-10-14T00:04:47.887621+0000","last_unstale":"2023-10-14T01:29:59.745287+0000","last_undegraded":"2023-10-14T01:29:59.745287+0000","last_fullsized":"2023-10-14T01:29:59.745287+0000","mapping_epoch":12,"log_start":"81'5500","ondisk_log_start":"81'5500","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'12464","last_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_deep_scrub":"85'12464","last_deep_scrub_stamp":"2023-10-14T01:10:25.234700+0000","last_clean_scrub_stamp":"2023-10-14T01:10:25.234700+0000","objects_scrubbed":11,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T11:41:09.898185+0000","scrub_duration":0.38186473500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":3927,"num_read_kb":3254,"num_write":95,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2940,"num_omap_keys":48,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"85'99","reported_seq":279,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-14T01:27:13.914771+0000","last_change":"2023-10-14T01:10:16.545738+0000","last_active":"2023-10-14T01:27:13.914771+0000","last_peered":"2023-10-14T01:27:13.914771+0000","last_clean":"2023-10-14T01:27:13.914771+0000","last_became_active":"2023-10-14T00:04:42.866562+0000","last_became_peered":"2023-10-14T00:04:42.866562+0000","last_unstale":"2023-10-14T01:27:13.914771+0000","last_undegraded":"2023-10-14T01:27:13.914771+0000","last_fullsized":"2023-10-14T01:27:13.914771+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'99","last_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_deep_scrub":"85'99","last_deep_scrub_stamp":"2023-10-14T01:10:16.545479+0000","last_clean_scrub_stamp":"2023-10-14T01:10:16.545479+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T12:37:32.662547+0000","scrub_duration":0.43422430899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":30267031571,"num_objects":15361,"num_object_clones":10240,"num_object_copies":30722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15361,"num_whiteouts":0,"num_read":2289281,"num_read_kb":9029612,"num_write":2285148,"num_write_kb":9140586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18719170560,"data_stored":21787197478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40083,"ondisk_log_size":40083,"up":8,"acting":8,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":3353,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":22331,"num_read_kb":18823,"num_write":673,"num_write_kb":363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":18003,"num_omap_keys":343,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":6706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39380,"internal_metadata":0},"log_size":48524,"ondisk_log_size":48524,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739396,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":11216832,"kb_used_data":9139320,"kb_used_omap":1426,"kb_used_meta":2076077,"kb_avail":83155008,"statfs":{"total":96636764160,"available":85150728192,"internally_reserved":0,"allocated":9358663680,"data_stored":10892132137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1461129,"internal_metadata":2125902967},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":0.30099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.20300000000000001,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":7.1040000000000001,"15min":7.1040000000000001},"last":0.437},{"interface":"front","average":{"1min":0.378,"5min":0.442,"15min":0.38100000000000001},"min":{"1min":0.22900000000000001,"5min":0.18099999999999999,"15min":0.16600000000000001},"max":{"1min":0.74099999999999999,"5min":3.835,"15min":3.835},"last":0.376}]},{"osd":1,"last update":"Sat Oct 14 01:29:52 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59399999999999997,"15min":0.53500000000000003},"min":{"1min":0.24199999999999999,"5min":0.23200000000000001,"15min":0.20399999999999999},"max":{"1min":3.02,"5min":7.1319999999999997,"15min":7.1859999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.60099999999999998,"15min":0.57099999999999995},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23699999999999999},"max":{"1min":0.61599999999999999,"5min":7.1479999999999997,"15min":7.8449999999999998},"last":0.40500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739396,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6076340,"kb_used_data":4583020,"kb_used_omap":704,"kb_used_meta":1492607,"kb_avail":88295500,"statfs":{"total":96636764160,"available":90414592000,"internally_reserved":0,"allocated":4693012480,"data_stored":5460060840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":721391,"internal_metadata":1528430097},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 01:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.505,"15min":0.52500000000000002},"min":{"1min":0.18099999999999999,"5min":0.17599999999999999,"15min":0.158},"max":{"1min":4.3479999999999999,"5min":4.3550000000000004,"15min":7.2809999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.27900000000000003,"5min":0.57299999999999995,"15min":0.54700000000000004},"min":{"1min":0.19,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.42499999999999999,"5min":9.2799999999999994,"15min":9.2799999999999994},"last":0.629}]},{"osd":2,"last update":"Sat Oct 14 01:29:34 2023","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.20799999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":0.40100000000000002,"5min":11.965999999999999,"15min":11.965999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.64600000000000002,"15min":0.56000000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.23899999999999999},"max":{"1min":0.42599999999999999,"5min":12.000999999999999,"15min":12.000999999999999},"last":0.76200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5991496,"kb_used_data":4559744,"kb_used_omap":753,"kb_used_meta":1430990,"kb_avail":88380344,"statfs":{"total":96636764160,"available":90501472256,"internally_reserved":0,"allocated":4669177856,"data_stored":5434676955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":771339,"internal_metadata":1465334517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 01:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.35799999999999998,"15min":0.41999999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17399999999999999},"max":{"1min":1.617,"5min":2.5259999999999998,"15min":5.8600000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.219,"5min":0.187,"15min":0.187},"max":{"1min":3.7360000000000002,"5min":6.2320000000000002,"15min":6.8369999999999997},"last":0.442}]},{"osd":2,"last update":"Sat Oct 14 01:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.442,"15min":0.625},"min":{"1min":0.21099999999999999,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":5.5380000000000003,"5min":5.5380000000000003,"15min":31.103000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.376,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.219,"15min":0.20399999999999999},"max":{"1min":3.5289999999999999,"5min":3.5289999999999999,"15min":30.234000000000002},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14024,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17836,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7520,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4667961344,"data_stored":5434146816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4692029440,"data_stored":5459857427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9359179776,"data_stored":10893193235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T01:30:02.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-14T01:30:02.719 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:30:02.717+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a2d1e10 2023-10-14T01:30:02.721 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:30:02.717+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d64e0 2023-10-14T01:30:02.736 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:30:02.733+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6820 2023-10-14T01:30:02.752 INFO:tasks.ceph.osd.1.smithi105.stderr:2023-10-14T01:30:02.749+0000 7f5baaa31640 -1 reset not still connected to 0x562d3a5d6a90 2023-10-14T01:30:02.775 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-14T01:30:02.776 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-14T01:30:02.776 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-14T01:30:02.777 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:30:06.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-14T01:30:08.879 INFO:tasks.ceph.osd.0:Stopped 2023-10-14T01:30:08.880 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-14T01:30:08.880 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:30:11.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-14T01:30:11.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-14T01:30:13.798 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T01:30:13.793+0000 7f34d0d24640 -1 reset not still connected to 0x55a91d49ed00 2023-10-14T01:30:13.799 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T01:30:13.793+0000 7f34d0d24640 -1 reset not still connected to 0x55a92a10bba0 2023-10-14T01:30:13.799 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T01:30:13.793+0000 7f34d0d24640 -1 reset not still connected to 0x55a92c9e11e0 2023-10-14T01:30:13.800 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T01:30:13.793+0000 7f34d0d24640 -1 reset not still connected to 0x55a92d14cf70 2023-10-14T01:30:13.800 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T01:30:13.793+0000 7f34d0d24640 -1 reset not still connected to 0x55a934e25ba0 2023-10-14T01:30:13.800 INFO:tasks.ceph.osd.2.smithi105.stderr:2023-10-14T01:30:13.793+0000 7f34d0d24640 -1 reset not still connected to 0x55a942d9dee0 2023-10-14T01:30:14.982 INFO:tasks.ceph.osd.1:Stopped 2023-10-14T01:30:14.983 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-14T01:30:14.983 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:30:16.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-14T01:30:16.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-14T01:30:21.085 INFO:tasks.ceph.osd.2:Stopped 2023-10-14T01:30:21.086 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-14T01:30:21.087 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-14T01:30:21.087 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:30:21.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2023-10-14T01:30:21.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-14T01:30:26.972 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2023-10-14T01:30:27.189 INFO:tasks.ceph.mgr.x:Stopped 2023-10-14T01:30:27.190 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-14T01:30:27.191 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-14T01:30:27.191 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T01:30:31.973 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-10-14T01:30:31.973 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2023-10-14T01:30:33.293 INFO:tasks.ceph.mon.a:Stopped 2023-10-14T01:30:33.293 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-14T01:30:33.294 DEBUG:teuthology.orchestra.run.smithi105:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-14T01:30:33.372 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T01:30:33.372 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-14T01:30:34.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T01:30:34.732 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-14T01:30:35.117 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T01:30:35.118 DEBUG:teuthology.orchestra.run.smithi105:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-14T01:30:35.525 INFO:tasks.ceph:Archiving mon data... 2023-10-14T01:30:35.528 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274/data/mon.a.tgz 2023-10-14T01:30:35.528 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2023-10-14T01:30:35.548 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.QV6BBSshbi 2023-10-14T01:30:35.549 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QV6BBSshbi 2023-10-14T01:30:35.690 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.QV6BBSshbi 2023-10-14T01:30:35.777 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.QV6BBSshbi is 451KB 2023-10-14T01:30:35.841 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.QV6BBSshbi 2023-10-14T01:30:35.847 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-14T01:30:35.848 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T01:30:35.948 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-14T01:30:35.949 DEBUG:teuthology.orchestra.run.smithi105:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T01:30:35.979 INFO:teuthology.orchestra.run.smithi105.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T01:30:35.991 INFO:tasks.ceph:Archiving crash dumps... 2023-10-14T01:30:35.993 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274/remote/smithi105/crash 2023-10-14T01:30:35.995 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T01:30:36.044 INFO:tasks.ceph:Compressing logs... 2023-10-14T01:30:36.045 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T01:30:36.118 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49916.log 2023-10-14T01:30:36.119 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47123.log 2023-10-14T01:30:36.120 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49916.log.gz 2023-10-14T01:30:36.121 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54786.log 2023-10-14T01:30:36.121 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51037.log 2023-10-14T01:30:36.121 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47123.log: /var/log/ceph/ceph-client.admin.54786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45884.log 2023-10-14T01:30:36.122 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54786.log.gz 2023-10-14T01:30:36.122 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2023-10-14T01:30:36.123 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51037.log.gz 2023-10-14T01:30:36.123 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49946.log 2023-10-14T01:30:36.128 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50312.log 2023-10-14T01:30:36.128 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40057.log: /var/log/ceph/ceph-client.admin.49946.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49946.log.gz 2023-10-14T01:30:36.129 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54310.log 2023-10-14T01:30:36.129 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40182.log 2023-10-14T01:30:36.129 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50312.log.gz 2023-10-14T01:30:36.130 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54310.log.gz 2023-10-14T01:30:36.130 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2023-10-14T01:30:36.130 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45011.log 2023-10-14T01:30:36.130 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41878.log 2023-10-14T01:30:36.131 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52042.log: /var/log/ceph/ceph-client.admin.45011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2023-10-14T01:30:36.131 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55614.log 2023-10-14T01:30:36.131 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55025.log 2023-10-14T01:30:36.132 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41878.log: /var/log/ceph/ceph-client.admin.55614.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.55614.log.gz /var/log/ceph/ceph-client.admin.51102.log 2023-10-14T01:30:36.132 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.132 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55322.log 2023-10-14T01:30:36.133 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55025.log.gz 2023-10-14T01:30:36.133 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41084.log 2023-10-14T01:30:36.133 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51102.log.gz 2023-10-14T01:30:36.133 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44580.log 2023-10-14T01:30:36.134 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55322.log.gz 2023-10-14T01:30:36.134 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43462.log 2023-10-14T01:30:36.134 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2023-10-14T01:30:36.135 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44580.log: /var/log/ceph/ceph-client.admin.43462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44705.log 2023-10-14T01:30:36.135 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46393.log 2023-10-14T01:30:36.135 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45442.log 2023-10-14T01:30:36.135 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47606.log 2023-10-14T01:30:36.136 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48215.log 2023-10-14T01:30:36.136 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45563.log 2023-10-14T01:30:36.136 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47606.log: /var/log/ceph/ceph-client.admin.48215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43971.log 2023-10-14T01:30:36.136 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44091.log 2023-10-14T01:30:36.205 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45318.log 2023-10-14T01:30:36.206 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43971.log: /var/log/ceph/ceph-client.admin.44091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47183.log 2023-10-14T01:30:36.206 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52802.log 2023-10-14T01:30:36.206 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46880.log 2023-10-14T01:30:36.206 INFO:teuthology.orchestra.run.smithi105.stderr:gzip/var/log/ceph/ceph-client.admin.52802.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42895.log 2023-10-14T01:30:36.207 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52802.log.gz 2023-10-14T01:30:36.207 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39145.log 2023-10-14T01:30:36.207 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47123.log.gz 2023-10-14T01:30:36.208 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46393.log.gz 2023-10-14T01:30:36.208 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47606.log.gz 2023-10-14T01:30:36.208 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45884.log.gz 2023-10-14T01:30:36.208 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46880.log.gz 2023-10-14T01:30:36.209 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45442.log.gz 2023-10-14T01:30:36.209 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48215.log.gz 2023-10-14T01:30:36.209 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45563.log.gz 2023-10-14T01:30:36.210 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45318.log.gz 2023-10-14T01:30:36.210 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42895.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.47183.log.gz --verbose -- 2023-10-14T01:30:36.210 INFO:teuthology.orchestra.run.smithi105.stderr: /var/log/ceph/ceph-client.admin.47303.log 2023-10-14T01:30:36.211 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 2023-10-14T01:30:36.211 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39145.log.gz 2023-10-14T01:30:36.211 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40273.log 2023-10-14T01:30:36.212 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47303.log.gz 2023-10-14T01:30:36.212 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50977.log 2023-10-14T01:30:36.212 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54390.log 2023-10-14T01:30:36.212 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50220.log 2023-10-14T01:30:36.213 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50977.log.gz 2023-10-14T01:30:36.213 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40182.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gz 2023-10-14T01:30:36.213 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.214 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54390.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43971.log.gz 2023-10-14T01:30:36.214 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54390.log.gz -- replaced with /var/log/ceph/ceph-client.admin.41878.log.gz 2023-10-14T01:30:36.214 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.215 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41084.log.gz 2023-10-14T01:30:36.215 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43462.log.gz -- replaced with /var/log/ceph/ceph-client.admin.45011.log.gz 2023-10-14T01:30:36.215 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.216 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42895.log.gz 2023-10-14T01:30:36.216 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2023-10-14T01:30:36.216 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42697.log 2023-10-14T01:30:36.217 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44580.log.gz 2023-10-14T01:30:36.217 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44705.log.gz 0.0% 2023-10-14T01:30:36.217 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2023-10-14T01:30:36.217 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44091.log.gz 2023-10-14T01:30:36.218 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2023-10-14T01:30:36.218 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40884.log 2023-10-14T01:30:36.218 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50220.log.gz 2023-10-14T01:30:36.218 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40256.log 2023-10-14T01:30:36.219 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42697.log.gz 2023-10-14T01:30:36.219 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40310.log 2023-10-14T01:30:36.219 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40884.log.gz 2023-10-14T01:30:36.219 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40256.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-14T01:30:36.220 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40256.log.gz 2023-10-14T01:30:36.220 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51587.log 2023-10-14T01:30:36.220 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40310.log.gz 2023-10-14T01:30:36.221 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55269.log 2023-10-14T01:30:36.221 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46971.log 2023-10-14T01:30:36.221 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51587.log.gz 2023-10-14T01:30:36.221 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53883.log 2023-10-14T01:30:36.222 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55269.log.gz 2023-10-14T01:30:36.222 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39008.log 2023-10-14T01:30:36.222 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46971.log.gz 2023-10-14T01:30:36.223 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53883.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.53883.log.gz 2023-10-14T01:30:36.223 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47153.log 2023-10-14T01:30:36.223 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44031.log 2023-10-14T01:30:36.223 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39008.log.gz 2023-10-14T01:30:36.223 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48944.log 2023-10-14T01:30:36.224 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47153.log.gz 2023-10-14T01:30:36.224 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48517.log 2023-10-14T01:30:36.224 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44031.log.gz 2023-10-14T01:30:36.224 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48944.log.gz 2023-10-14T01:30:36.225 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53804.log 2023-10-14T01:30:36.225 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48335.log 2023-10-14T01:30:36.225 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48517.log.gz 2023-10-14T01:30:36.225 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53323.log 2023-10-14T01:30:36.226 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53804.log.gz 2023-10-14T01:30:36.226 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46513.log 2023-10-14T01:30:36.226 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48335.log.gz 2023-10-14T01:30:36.227 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53323.log.gz 2023-10-14T01:30:36.227 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2023-10-14T01:30:36.227 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41592.log 2023-10-14T01:30:36.227 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46513.log.gz 2023-10-14T01:30:36.227 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2023-10-14T01:30:36.228 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2023-10-14T01:30:36.228 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44520.log 2023-10-14T01:30:36.228 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41592.log.gz 2023-10-14T01:30:36.228 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54947.log 2023-10-14T01:30:36.229 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2023-10-14T01:30:36.229 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52588.log 2023-10-14T01:30:36.229 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44520.log.gz 2023-10-14T01:30:36.229 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53409.log 2023-10-14T01:30:36.230 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54947.log.gz 2023-10-14T01:30:36.230 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2023-10-14T01:30:36.230 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52588.log.gz 2023-10-14T01:30:36.231 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53409.log.gz 2023-10-14T01:30:36.231 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50036.log 2023-10-14T01:30:36.231 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49491.log 2023-10-14T01:30:36.231 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2023-10-14T01:30:36.232 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50036.log.gz 2023-10-14T01:30:36.232 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50764.log 2023-10-14T01:30:36.232 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50492.log 2023-10-14T01:30:36.233 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49491.log.gz 2023-10-14T01:30:36.233 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45995.log 2023-10-14T01:30:36.233 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50764.log.gz 2023-10-14T01:30:36.233 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41500.log 2023-10-14T01:30:36.234 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50492.log.gz 2023-10-14T01:30:36.234 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55694.log 2023-10-14T01:30:36.234 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45995.log.gz 2023-10-14T01:30:36.235 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41500.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41500.log.gz -5 2023-10-14T01:30:36.235 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52163.log 2023-10-14T01:30:36.235 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55238.log 2023-10-14T01:30:36.235 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55694.log.gz 2023-10-14T01:30:36.236 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49703.log 2023-10-14T01:30:36.236 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52163.log.gz 2023-10-14T01:30:36.236 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55238.log.gz 2023-10-14T01:30:36.237 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55454.log 2023-10-14T01:30:36.237 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49551.log 2023-10-14T01:30:36.237 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49703.log.gz 2023-10-14T01:30:36.237 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45258.log 2023-10-14T01:30:36.238 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55454.log.gz 2023-10-14T01:30:36.238 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43066.log 2023-10-14T01:30:36.238 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49551.log.gz 2023-10-14T01:30:36.239 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40958.log 2023-10-14T01:30:36.239 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45258.log.gz 2023-10-14T01:30:36.239 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2023-10-14T01:30:36.239 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43066.log.gz 2023-10-14T01:30:36.240 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44433.log 2023-10-14T01:30:36.240 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2023-10-14T01:30:36.240 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54227.log 2023-10-14T01:30:36.241 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2023-10-14T01:30:36.241 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2023-10-14T01:30:36.241 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44433.log.gz 2023-10-14T01:30:36.242 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46574.log 2023-10-14T01:30:36.242 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54227.log.gz 2023-10-14T01:30:36.242 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39449.log 2023-10-14T01:30:36.242 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz 2023-10-14T01:30:36.243 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51007.log 2023-10-14T01:30:36.243 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46574.log.gz 2023-10-14T01:30:36.243 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53487.log 2023-10-14T01:30:36.244 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39449.log.gz 2023-10-14T01:30:36.244 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2023-10-14T01:30:36.244 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51007.log.gz 2023-10-14T01:30:36.244 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40094.log 2023-10-14T01:30:36.245 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53487.log.gz 2023-10-14T01:30:36.245 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41942.log 2023-10-14T01:30:36.245 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2023-10-14T01:30:36.245 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48671.log 2023-10-14T01:30:36.246 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40094.log.gz 2023-10-14T01:30:36.246 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53341.log 2023-10-14T01:30:36.246 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41942.log.gz 2023-10-14T01:30:36.247 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43802.log 2023-10-14T01:30:36.247 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48671.log.gz 2023-10-14T01:30:36.247 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2023-10-14T01:30:36.247 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53341.log.gz 2023-10-14T01:30:36.248 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43802.log.gzgzip 2023-10-14T01:30:36.248 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49004.log 2023-10-14T01:30:36.248 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47093.log 2023-10-14T01:30:36.248 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2023-10-14T01:30:36.249 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41634.log 2023-10-14T01:30:36.249 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49004.log.gz 2023-10-14T01:30:36.249 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47093.log.gz 2023-10-14T01:30:36.250 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47697.log 2023-10-14T01:30:36.250 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50402.log 2023-10-14T01:30:36.250 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41634.log.gz 2023-10-14T01:30:36.251 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51830.log 2023-10-14T01:30:36.251 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47697.log.gz 2023-10-14T01:30:36.251 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50402.log.gz 2023-10-14T01:30:36.251 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53195.log 2023-10-14T01:30:36.252 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46186.log 2023-10-14T01:30:36.252 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51830.log.gz 2023-10-14T01:30:36.252 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2023-10-14T01:30:36.252 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53195.log.gz 2023-10-14T01:30:36.253 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2023-10-14T01:30:36.253 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46186.log.gz 2023-10-14T01:30:36.253 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2023-10-14T01:30:36.254 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50126.log 2023-10-14T01:30:36.254 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53268.log 2023-10-14T01:30:36.254 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2023-10-14T01:30:36.254 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51526.log 2023-10-14T01:30:36.255 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50126.log.gz 2023-10-14T01:30:36.255 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55507.log 2023-10-14T01:30:36.255 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53268.log.gz 2023-10-14T01:30:36.256 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5/var/log/ceph/ceph-client.admin.51526.log: --verbose -- /var/log/ceph/ceph-client.admin.41196.log 2023-10-14T01:30:36.256 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51526.log.gz 2023-10-14T01:30:36.256 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44001.log 2023-10-14T01:30:36.256 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55507.log.gz 2023-10-14T01:30:36.257 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41196.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41196.log.gz -5 2023-10-14T01:30:36.257 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46453.log 2023-10-14T01:30:36.257 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44001.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-14T01:30:36.257 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44001.log.gz 2023-10-14T01:30:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50157.log 2023-10-14T01:30:36.258 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46453.log.gz 2023-10-14T01:30:36.258 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41970.log 2023-10-14T01:30:36.259 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49885.log 2023-10-14T01:30:36.259 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50157.log.gz 2023-10-14T01:30:36.259 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51225.log 2023-10-14T01:30:36.259 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41970.log.gz 2023-10-14T01:30:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40992.log 2023-10-14T01:30:36.260 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49885.log.gz 2023-10-14T01:30:36.260 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51225.log.gz 2023-10-14T01:30:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52345.log 2023-10-14T01:30:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43110.log 2023-10-14T01:30:36.261 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40992.log.gz 2023-10-14T01:30:36.261 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47484.log 2023-10-14T01:30:36.262 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52345.log.gz 2023-10-14T01:30:36.262 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40776.log 2023-10-14T01:30:36.262 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43110.log.gz 2023-10-14T01:30:36.262 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50187.log 2023-10-14T01:30:36.263 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47484.log.gz 2023-10-14T01:30:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49400.log 2023-10-14T01:30:36.263 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40776.log.gz 2023-10-14T01:30:36.263 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47062.log 2023-10-14T01:30:36.264 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50187.log.gz 2023-10-14T01:30:36.264 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45390.log 2023-10-14T01:30:36.264 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49400.log.gz 2023-10-14T01:30:36.264 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44490.log 2023-10-14T01:30:36.265 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47062.log.gz 2023-10-14T01:30:36.265 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46089.log 2023-10-14T01:30:36.265 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45390.log.gz 2023-10-14T01:30:36.265 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44826.log 2023-10-14T01:30:36.266 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44490.log.gz 2023-10-14T01:30:36.266 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55080.log 2023-10-14T01:30:36.266 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46089.log.gz 2023-10-14T01:30:36.267 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43405.log 2023-10-14T01:30:36.267 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44826.log.gz 2023-10-14T01:30:36.267 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55080.log: /var/log/ceph/ceph-client.admin.34653.log 2023-10-14T01:30:36.267 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55080.log.gz 2023-10-14T01:30:36.268 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47423.log 2023-10-14T01:30:36.268 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43405.log.gz 2023-10-14T01:30:36.268 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34653.log.gz 2023-10-14T01:30:36.269 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52496.log 2023-10-14T01:30:36.269 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41710.log 2023-10-14T01:30:36.269 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47423.log.gz 2023-10-14T01:30:36.269 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52496.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.52496.log.gz --verbose 2023-10-14T01:30:36.270 INFO:teuthology.orchestra.run.smithi105.stderr: -- /var/log/ceph/ceph-client.admin.43494.log 2023-10-14T01:30:36.270 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40381.log 2023-10-14T01:30:36.270 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41710.log.gz 2023-10-14T01:30:36.270 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43494.log.gz 2023-10-14T01:30:36.271 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42427.log 2023-10-14T01:30:36.271 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2023-10-14T01:30:36.271 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40381.log.gz 2023-10-14T01:30:36.272 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42427.log.gz 2023-10-14T01:30:36.272 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55559.log 2023-10-14T01:30:36.272 INFO:teuthology.orchestra.run.smithi105.stderr:gzip/var/log/ceph/ceph-client.admin.54524.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-14T01:30:36.272 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2023-10-14T01:30:36.273 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55559.log.gz 2023-10-14T01:30:36.273 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2023-10-14T01:30:36.273 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose --/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.40398.log 2023-10-14T01:30:36.273 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2023-10-14T01:30:36.274 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47944.log 2023-10-14T01:30:36.274 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40398.log.gz 2023-10-14T01:30:36.274 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2023-10-14T01:30:36.275 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55533.log 2023-10-14T01:30:36.275 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47944.log.gz 2023-10-14T01:30:36.275 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2023-10-14T01:30:36.276 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2023-10-14T01:30:36.276 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2023-10-14T01:30:36.276 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55533.log.gz 2023-10-14T01:30:36.276 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55401.log 2023-10-14T01:30:36.277 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2023-10-14T01:30:36.277 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44460.log 2023-10-14T01:30:36.277 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2023-10-14T01:30:36.277 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48974.log 2023-10-14T01:30:36.278 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55401.log.gz 2023-10-14T01:30:36.278 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2023-10-14T01:30:36.278 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44460.log.gz 2023-10-14T01:30:36.278 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48974.log.gz 2023-10-14T01:30:36.279 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40975.log 2023-10-14T01:30:36.279 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54254.log 2023-10-14T01:30:36.279 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2023-10-14T01:30:36.280 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40975.log.gz 2023-10-14T01:30:36.280 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54092.log 2023-10-14T01:30:36.280 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51466.log 2023-10-14T01:30:36.280 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54254.log.gz 2023-10-14T01:30:36.281 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54092.log.gz 2023-10-14T01:30:36.281 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44163.log 2023-10-14T01:30:36.281 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39146.log 2023-10-14T01:30:36.281 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51466.log.gz 2023-10-14T01:30:36.282 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49365.log 2023-10-14T01:30:36.282 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44163.log.gz 2023-10-14T01:30:36.282 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48456.log 2023-10-14T01:30:36.283 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39146.log.gz 2023-10-14T01:30:36.283 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47333.log 2023-10-14T01:30:36.283 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49365.log.gz 2023-10-14T01:30:36.283 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43353.log 2023-10-14T01:30:36.284 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48456.log.gz 2023-10-14T01:30:36.284 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49611.log 2023-10-14T01:30:36.284 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47333.log.gz 2023-10-14T01:30:36.284 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40813.log 2023-10-14T01:30:36.285 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43353.log.gz 2023-10-14T01:30:36.285 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50796.log 2023-10-14T01:30:36.285 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49611.log.gz 2023-10-14T01:30:36.286 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40813.log.gz 2023-10-14T01:30:36.286 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51557.log 2023-10-14T01:30:36.286 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51345.log 2023-10-14T01:30:36.287 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2023-10-14T01:30:36.287 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51557.log.gz 2023-10-14T01:30:36.287 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51921.log 2023-10-14T01:30:36.288 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51345.log.gz 2023-10-14T01:30:36.288 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43526.log 2023-10-14T01:30:36.288 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz 2023-10-14T01:30:36.288 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2023-10-14T01:30:36.289 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51921.log.gz 2023-10-14T01:30:36.289 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53231.log 2023-10-14T01:30:36.289 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43526.log.gz 2023-10-14T01:30:36.289 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42334.log 2023-10-14T01:30:36.290 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% 29.2% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2023-10-14T01:30:36.290 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50796.log.gz 2023-10-14T01:30:36.290 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2023-10-14T01:30:36.290 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53231.log.gz 2023-10-14T01:30:36.291 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48486.log 2023-10-14T01:30:36.291 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42334.log.gz 2023-10-14T01:30:36.291 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2023-10-14T01:30:36.292 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2023-10-14T01:30:36.292 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53539.log 2023-10-14T01:30:36.292 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48486.log.gz 2023-10-14T01:30:36.292 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48035.log 2023-10-14T01:30:36.293 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2023-10-14T01:30:36.293 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42234.log 2023-10-14T01:30:36.293 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53539.log.gz 2023-10-14T01:30:36.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2023-10-14T01:30:36.294 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48035.log.gz 2023-10-14T01:30:36.294 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49335.log 2023-10-14T01:30:36.294 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42234.log.gz 2023-10-14T01:30:36.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2023-10-14T01:30:36.295 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2023-10-14T01:30:36.295 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2023-10-14T01:30:36.295 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49335.log.gz 2023-10-14T01:30:36.296 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49641.log 2023-10-14T01:30:36.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2023-10-14T01:30:36.296 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55668.log 2023-10-14T01:30:36.297 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2023-10-14T01:30:36.297 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53030.log 2023-10-14T01:30:36.297 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49641.log.gz 2023-10-14T01:30:36.297 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47363.log 2023-10-14T01:30:36.298 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55668.log.gz 2023-10-14T01:30:36.298 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53030.log.gz 2023-10-14T01:30:36.298 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34511.log 2023-10-14T01:30:36.299 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53213.log 2023-10-14T01:30:36.299 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47363.log.gz 2023-10-14T01:30:36.299 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42823.log 2023-10-14T01:30:36.299 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34511.log.gz 2023-10-14T01:30:36.300 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43006.log 2023-10-14T01:30:36.300 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53213.log.gz 2023-10-14T01:30:36.300 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2023-10-14T01:30:36.300 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42823.log.gz 2023-10-14T01:30:36.301 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50644.log 2023-10-14T01:30:36.301 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43006.log.gz 2023-10-14T01:30:36.301 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2023-10-14T01:30:36.301 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz 2023-10-14T01:30:36.302 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44550.log 2023-10-14T01:30:36.302 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50644.log.gz 2023-10-14T01:30:36.302 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2023-10-14T01:30:36.303 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2023-10-14T01:30:36.303 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53909.log 2023-10-14T01:30:36.303 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44550.log.gz 2023-10-14T01:30:36.303 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46604.log 2023-10-14T01:30:36.304 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2023-10-14T01:30:36.304 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46940.log 2023-10-14T01:30:36.304 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53909.log.gz 2023-10-14T01:30:36.304 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2023-10-14T01:30:36.305 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46604.log.gz 2023-10-14T01:30:36.305 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40165.log 2023-10-14T01:30:36.305 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46940.log.gz 2023-10-14T01:30:36.305 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52224.log 2023-10-14T01:30:36.306 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2023-10-14T01:30:36.306 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2023-10-14T01:30:36.306 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40165.log.gz 2023-10-14T01:30:36.307 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53158.log 2023-10-14T01:30:36.307 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2023-10-14T01:30:36.307 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43189.log 2023-10-14T01:30:36.307 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53103.log 2023-10-14T01:30:36.308 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53935.log 2023-10-14T01:30:36.308 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52618.log 2023-10-14T01:30:36.308 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49065.log 2023-10-14T01:30:36.308 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55134.log 2023-10-14T01:30:36.309 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54283.log 2023-10-14T01:30:36.309 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47001.log 2023-10-14T01:30:36.309 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55053.log 2023-10-14T01:30:36.309 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2023-10-14T01:30:36.310 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48852.log 2023-10-14T01:30:36.310 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41012.log 2023-10-14T01:30:36.310 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47273.log 2023-10-14T01:30:36.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2023-10-14T01:30:36.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42264.log 2023-10-14T01:30:36.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52012.log 2023-10-14T01:30:36.311 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51375.log 2023-10-14T01:30:36.312 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50704.log 2023-10-14T01:30:36.312 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48095.log 2023-10-14T01:30:36.312 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46161.log 2023-10-14T01:30:36.313 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49976.log 2023-10-14T01:30:36.313 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50887.log 2023-10-14T01:30:36.313 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41248.log 2023-10-14T01:30:36.313 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45840.log 2023-10-14T01:30:36.314 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45964.log 2023-10-14T01:30:36.314 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52254.log 2023-10-14T01:30:36.314 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47763.log 2023-10-14T01:30:36.315 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46787.log 2023-10-14T01:30:36.315 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2023-10-14T01:30:36.315 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51709.log 2023-10-14T01:30:36.315 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45665.log 2023-10-14T01:30:36.316 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45414.log 2023-10-14T01:30:36.316 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50066.log 2023-10-14T01:30:36.316 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2023-10-14T01:30:36.316 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44923.log 2023-10-14T01:30:36.317 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53383.log 2023-10-14T01:30:36.317 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55186.log 2023-10-14T01:30:36.317 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2023-10-14T01:30:36.318 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55374.log 2023-10-14T01:30:36.318 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54118.log 2023-10-14T01:30:36.318 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54170.log 2023-10-14T01:30:36.318 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2023-10-14T01:30:36.319 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45593.log 2023-10-14T01:30:36.319 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53461.log 2023-10-14T01:30:36.319 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2023-10-14T01:30:36.320 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46423.log 2023-10-14T01:30:36.320 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45145.log 2023-10-14T01:30:36.320 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44187.log 2023-10-14T01:30:36.320 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42759.log 2023-10-14T01:30:36.321 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47914.log 2023-10-14T01:30:36.321 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2023-10-14T01:30:36.321 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49245.log 2023-10-14T01:30:36.321 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2023-10-14T01:30:36.322 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42174.log 2023-10-14T01:30:36.322 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52133.log 2023-10-14T01:30:36.322 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2023-10-14T01:30:36.323 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39983.log 2023-10-14T01:30:36.323 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43771.log 2023-10-14T01:30:36.323 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51255.log 2023-10-14T01:30:36.323 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48821.log 2023-10-14T01:30:36.324 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2023-10-14T01:30:36.324 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48731.log 2023-10-14T01:30:36.324 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53723.log 2023-10-14T01:30:36.324 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55718.log 2023-10-14T01:30:36.325 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54867.log 2023-10-14T01:30:36.325 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47454.log 2023-10-14T01:30:36.325 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53988.log 2023-10-14T01:30:36.325 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2023-10-14T01:30:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-14T01:30:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48914.log 2023-10-14T01:30:36.326 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49035.log 2023-10-14T01:30:36.327 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52648.log 2023-10-14T01:30:36.327 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45533.log 2023-10-14T01:30:36.327 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50826.log 2023-10-14T01:30:36.327 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2023-10-14T01:30:36.328 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54196.log 2023-10-14T01:30:36.427 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42793.log 2023-10-14T01:30:36.428 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47032.log 2023-10-14T01:30:36.428 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48761.log 2023-10-14T01:30:36.428 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45169.log 2023-10-14T01:30:36.429 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49125.log 2023-10-14T01:30:36.429 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39060.log 2023-10-14T01:30:36.429 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49095.log 2023-10-14T01:30:36.430 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42976.log 2023-10-14T01:30:36.430 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51315.log 2023-10-14T01:30:36.430 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47393.log 2023-10-14T01:30:36.430 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54498.log 2023-10-14T01:30:36.431 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39398.log 2023-10-14T01:30:36.431 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46243.log 2023-10-14T01:30:36.431 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54973.log 2023-10-14T01:30:36.432 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45912.log 2023-10-14T01:30:36.432 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47666.log 2023-10-14T01:30:36.432 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2023-10-14T01:30:36.433 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42867.log 2023-10-14T01:30:36.433 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2023-10-14T01:30:36.433 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55428.log 2023-10-14T01:30:36.433 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52285.log 2023-10-14T01:30:36.434 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49185.log 2023-10-14T01:30:36.434 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46634.log 2023-10-14T01:30:36.434 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44061.log 2023-10-14T01:30:36.435 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49521.log 2023-10-14T01:30:36.435 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42618.log 2023-10-14T01:30:36.435 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52375.log 2023-10-14T01:30:36.436 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40850.log 2023-10-14T01:30:36.436 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54920.log 2023-10-14T01:30:36.436 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52938.log 2023-10-14T01:30:36.437 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46303.log 2023-10-14T01:30:36.437 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54814.log 2023-10-14T01:30:36.437 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51285.log 2023-10-14T01:30:36.437 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43741.log 2023-10-14T01:30:36.438 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52405.log 2023-10-14T01:30:36.438 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52742.log 2023-10-14T01:30:36.438 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53750.log 2023-10-14T01:30:36.439 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47733.log 2023-10-14T01:30:36.439 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52193.log 2023-10-14T01:30:36.439 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40489.log 2023-10-14T01:30:36.439 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42375.log 2023-10-14T01:30:36.440 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30920.log 2023-10-14T01:30:36.440 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49155.log 2023-10-14T01:30:36.440 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-14T01:30:36.440 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2023-10-14T01:30:36.441 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42123.log 2023-10-14T01:30:36.441 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54894.log 2023-10-14T01:30:36.441 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50372.log 2023-10-14T01:30:36.442 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51647.log 2023-10-14T01:30:36.442 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44307.log 2023-10-14T01:30:36.442 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2023-10-14T01:30:36.442 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34629.log 2023-10-14T01:30:36.443 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54656.log 2023-10-14T01:30:36.443 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T01:30:36.443 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44626.log 2023-10-14T01:30:36.443 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54708.log 2023-10-14T01:30:36.444 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52224.log: /var/log/ceph/ceph-client.admin.53617.log: /var/log/ceph/ceph-client.admin.44626.log: /var/log/ceph/ceph-client.admin.53158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52224.log.gz 2023-10-14T01:30:36.444 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44626.log.gz 2023-10-14T01:30:36.444 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2023-10-14T01:30:36.445 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53158.log.gz 2023-10-14T01:30:36.445 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43432.log 2023-10-14T01:30:36.445 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40722.log: /var/log/ceph/ceph-client.admin.43189.log: /var/log/ceph/ceph-client.admin.53103.log: /var/log/ceph/ceph-client.admin.53935.log: /var/log/ceph/ceph-client.admin.54708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz 2023-10-14T01:30:36.445 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43189.log.gz 2023-10-14T01:30:36.446 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53103.log.gz 2023-10-14T01:30:36.446 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53935.log.gz 2023-10-14T01:30:36.446 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54708.log.gz 2023-10-14T01:30:36.447 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2023-10-14T01:30:36.447 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52618.log: /var/log/ceph/ceph-client.admin.49065.log: /var/log/ceph/ceph-client.admin.43432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52618.log.gz 2023-10-14T01:30:36.447 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49065.log.gz 2023-10-14T01:30:36.447 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43432.log.gz 2023-10-14T01:30:36.448 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0%/var/log/ceph/ceph-client.admin.54283.log: -- replaced with /var/log/ceph/ceph-client.admin.55134.log.gz 2023-10-14T01:30:36.448 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54283.log.gz 2023-10-14T01:30:36.448 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52678.log 2023-10-14T01:30:36.448 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47001.log.gz 2023-10-14T01:30:36.449 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55053.log.gz 2023-10-14T01:30:36.449 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46213.log: /var/log/ceph/ceph-client.admin.48852.log: /var/log/ceph/ceph-client.admin.41012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2023-10-14T01:30:36.449 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47576.log: /var/log/ceph/ceph-client.admin.47273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48852.log.gz 2023-10-14T01:30:36.449 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40526.log 2023-10-14T01:30:36.450 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41012.log.gz 2023-10-14T01:30:36.450 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz 2023-10-14T01:30:36.450 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47273.log.gz 2023-10-14T01:30:36.451 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44409.log: /var/log/ceph/ceph-client.admin.42264.log: /var/log/ceph/ceph-client.admin.52012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2023-10-14T01:30:36.451 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51375.log: /var/log/ceph/ceph-client.admin.52678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52012.log.gz 2023-10-14T01:30:36.451 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42264.log.gz 2023-10-14T01:30:36.452 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51375.log.gz 2023-10-14T01:30:36.452 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52678.log.gz 2023-10-14T01:30:36.452 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54014.log 2023-10-14T01:30:36.452 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50704.log: /var/log/ceph/ceph-client.admin.48095.log: /var/log/ceph/ceph-client.admin.46161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50704.log.gz 2023-10-14T01:30:36.453 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48095.log.gz 2023-10-14T01:30:36.453 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46161.log.gz 2023-10-14T01:30:36.453 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40526.log.gz 2023-10-14T01:30:36.454 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49976.log.gzgzip 2023-10-14T01:30:36.454 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45637.log 2023-10-14T01:30:36.454 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50887.log: /var/log/ceph/ceph-client.admin.41248.log: /var/log/ceph/ceph-client.admin.45840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50887.log.gz 2023-10-14T01:30:36.454 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41248.log.gz 2023-10-14T01:30:36.455 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45840.log.gz 2023-10-14T01:30:36.455 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45964.log.gz 2023-10-14T01:30:36.455 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2023-10-14T01:30:36.456 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52254.log: /var/log/ceph/ceph-client.admin.47763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54014.log.gz 2023-10-14T01:30:36.456 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46787.log: /var/log/ceph/ceph-client.admin.39221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52254.log.gz 2023-10-14T01:30:36.456 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47763.log.gz 2023-10-14T01:30:36.456 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46787.log.gz 2023-10-14T01:30:36.457 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2023-10-14T01:30:36.457 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51860.log 2023-10-14T01:30:36.457 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45637.log.gz 2023-10-14T01:30:36.458 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51709.log.gz 2023-10-14T01:30:36.458 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45665.log: /var/log/ceph/ceph-client.admin.45414.log: /var/log/ceph/ceph-client.admin.50066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2023-10-14T01:30:36.458 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45665.log.gz 2023-10-14T01:30:36.459 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45414.log.gz 2023-10-14T01:30:36.459 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50066.log.gz 2023-10-14T01:30:36.459 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41761.log: /var/log/ceph/ceph-client.admin.44923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2023-10-14T01:30:36.459 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44923.log.gz 2023-10-14T01:30:36.460 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53383.log: /var/log/ceph/ceph-client.admin.55186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2023-10-14T01:30:36.460 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51860.log.gz 2023-10-14T01:30:36.460 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48791.log: 0.0%/var/log/ceph/ceph-client.admin.41374.log: -- replaced with /var/log/ceph/ceph-client.admin.53383.log.gz 2023-10-14T01:30:36.461 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55186.log.gz 2023-10-14T01:30:36.461 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2023-10-14T01:30:36.461 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2023-10-14T01:30:36.461 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55374.log: /var/log/ceph/ceph-client.admin.40364.log: /var/log/ceph/ceph-client.admin.54118.log: /var/log/ceph/ceph-client.admin.54170.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.55374.log.gz -- 2023-10-14T01:30:36.462 INFO:teuthology.orchestra.run.smithi105.stderr: /var/log/ceph/ceph-client.admin.46817.log 2023-10-14T01:30:36.462 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 0.0% 2023-10-14T01:30:36.462 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54118.log.gz -- replaced with /var/log/ceph/ceph-client.admin.54170.log.gz 2023-10-14T01:30:36.463 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.463 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53831.log: /var/log/ceph/ceph-client.admin.45593.log: /var/log/ceph/ceph-client.admin.53461.log: /var/log/ceph/ceph-client.admin.41046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2023-10-14T01:30:36.463 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53461.log.gz 2023-10-14T01:30:36.464 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2023-10-14T01:30:36.464 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49764.log 2023-10-14T01:30:36.464 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz -- replaced with /var/log/ceph/ceph-client.admin.45593.log.gz 2023-10-14T01:30:36.465 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.465 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46423.log: /var/log/ceph/ceph-client.admin.45145.log: /var/log/ceph/ceph-client.admin.44187.log: /var/log/ceph/ceph-client.admin.42759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2023-10-14T01:30:36.465 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45145.log.gz 2023-10-14T01:30:36.465 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44187.log.gz 2023-10-14T01:30:36.466 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47914.log: /var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44337.log 2023-10-14T01:30:36.466 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46817.log.gz 2023-10-14T01:30:36.466 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47914.log.gz 2023-10-14T01:30:36.467 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0%/var/log/ceph/ceph-client.admin.39424.log: -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2023-10-14T01:30:36.467 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0%/var/log/ceph/ceph-client.admin.42174.log: -- replaced with /var/log/ceph/ceph-client.admin.49245.log.gz 2023-10-14T01:30:36.467 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2023-10-14T01:30:36.467 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42174.log.gz 2023-10-14T01:30:36.468 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52133.log.gz 2023-10-14T01:30:36.468 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49764.log.gz 2023-10-14T01:30:36.468 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40904.log: /var/log/ceph/ceph-client.admin.39983.log: /var/log/ceph/ceph-client.admin.43771.log: /var/log/ceph/ceph-client.admin.51255.log: /var/log/ceph/ceph-client.admin.48821.log: /var/log/ceph/ceph-client.admin.41132.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39983.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2023-10-14T01:30:36.468 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.469 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43771.log.gz 2023-10-14T01:30:36.469 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51255.log.gz 2023-10-14T01:30:36.469 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48821.log.gz 2023-10-14T01:30:36.470 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gz 2023-10-14T01:30:36.470 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48731.log: /var/log/ceph/ceph-client.admin.53723.log: /var/log/ceph/ceph-client.admin.55718.log: /var/log/ceph/ceph-client.admin.54867.log: /var/log/ceph/ceph-client.admin.47454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48731.log.gz 2023-10-14T01:30:36.470 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53723.log.gz 2023-10-14T01:30:36.471 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55718.log.gz 2023-10-14T01:30:36.471 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47454.log.gz 2023-10-14T01:30:36.471 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54867.log.gz 2023-10-14T01:30:36.471 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53988.log.gz 2023-10-14T01:30:36.472 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39294.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.48914.log: /var/log/ceph/ceph-client.admin.49035.log: /var/log/ceph/ceph-client.admin.52648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2023-10-14T01:30:36.472 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48914.log.gz 2023-10-14T01:30:36.472 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49035.log.gz 2023-10-14T01:30:36.473 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52648.log.gz 2023-10-14T01:30:36.473 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45533.log: /var/log/ceph/ceph-client.admin.50826.log: /var/log/ceph/ceph-client.admin.40597.log: /var/log/ceph/ceph-client.admin.54196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45533.log.gz 2023-10-14T01:30:36.473 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2023-10-14T01:30:36.473 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50826.log.gz 2023-10-14T01:30:36.474 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54196.log.gz 2023-10-14T01:30:36.474 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42793.log: /var/log/ceph/ceph-client.admin.47032.log: /var/log/ceph/ceph-client.admin.48761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42793.log.gz 2023-10-14T01:30:36.474 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45169.log: /var/log/ceph/ceph-client.admin.49125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47032.log.gz 2023-10-14T01:30:36.475 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48761.log.gz 2023-10-14T01:30:36.475 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45169.log.gz 2023-10-14T01:30:36.475 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49125.log.gz 2023-10-14T01:30:36.475 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39060.log: /var/log/ceph/ceph-client.admin.49095.log: /var/log/ceph/ceph-client.admin.42976.log: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.42759.log.gz 2023-10-14T01:30:36.476 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39060.log.gz 2023-10-14T01:30:36.476 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49095.log.gz 2023-10-14T01:30:36.476 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42976.log.gz 2023-10-14T01:30:36.476 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51315.log.gz 2023-10-14T01:30:36.477 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53249.log 2023-10-14T01:30:36.477 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47393.log: /var/log/ceph/ceph-client.admin.44337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42574.log 2023-10-14T01:30:36.477 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54498.log: /var/log/ceph/ceph-client.admin.39398.log: /var/log/ceph/ceph-client.admin.46243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47393.log.gz 2023-10-14T01:30:36.477 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44337.log.gz 2023-10-14T01:30:36.478 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54498.log.gz 2023-10-14T01:30:36.478 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39398.log.gz 2023-10-14T01:30:36.478 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46243.log.gz 2023-10-14T01:30:36.479 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42946.log 2023-10-14T01:30:36.479 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54973.log: /var/log/ceph/ceph-client.admin.45912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53249.log.gz 2023-10-14T01:30:36.479 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47666.log: /var/log/ceph/ceph-client.admin.52956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54973.log.gz 2023-10-14T01:30:36.480 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45912.log.gz 2023-10-14T01:30:36.480 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2023-10-14T01:30:36.480 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47666.log.gz 2023-10-14T01:30:36.480 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51981.log 2023-10-14T01:30:36.481 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42574.log.gz/var/log/ceph/ceph-client.admin.42867.log: 2023-10-14T01:30:36.481 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40290.log: /var/log/ceph/ceph-client.admin.55428.log: /var/log/ceph/ceph-client.admin.52285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42867.log.gz 2023-10-14T01:30:36.481 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2023-10-14T01:30:36.481 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55428.log.gz 2023-10-14T01:30:36.482 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52285.log.gz 2023-10-14T01:30:36.482 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49305.log 2023-10-14T01:30:36.482 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42946.log: 0.0%/var/log/ceph/ceph-client.admin.49185.log: -- replaced with /var/log/ceph/ceph-client.admin.42946.log.gz 2023-10-14T01:30:36.482 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46634.log: /var/log/ceph/ceph-client.admin.44061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49185.log.gz 2023-10-14T01:30:36.483 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46634.log.gz 2023-10-14T01:30:36.483 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44061.log.gz 2023-10-14T01:30:36.483 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53305.log 2023-10-14T01:30:36.483 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51981.log.gz 2023-10-14T01:30:36.484 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42618.log: /var/log/ceph/ceph-client.admin.52375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49521.log.gz 2023-10-14T01:30:36.484 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42618.log.gz 2023-10-14T01:30:36.484 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52375.log.gz 2023-10-14T01:30:36.484 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45043.log 2023-10-14T01:30:36.485 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49305.log.gz 2023-10-14T01:30:36.485 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40850.log: /var/log/ceph/ceph-client.admin.54920.log: /var/log/ceph/ceph-client.admin.52938.log: /var/log/ceph/ceph-client.admin.46303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40850.log.gz 2023-10-14T01:30:36.485 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54920.log.gz 2023-10-14T01:30:36.486 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52938.log.gz 2023-10-14T01:30:36.486 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53305.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46303.log.gz /var/log/ceph/ceph-client.admin.45810.log 2023-10-14T01:30:36.486 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.486 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53305.log.gz 2023-10-14T01:30:36.487 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54814.log: /var/log/ceph/ceph-client.admin.51285.log: /var/log/ceph/ceph-client.admin.43741.log: /var/log/ceph/ceph-client.admin.52405.log: /var/log/ceph/ceph-client.admin.45043.log: /var/log/ceph/ceph-client.admin.53750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54814.log.gz 2023-10-14T01:30:36.487 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43741.log.gz 2023-10-14T01:30:36.487 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51285.log.gz 2023-10-14T01:30:36.488 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52405.log.gz 2023-10-14T01:30:36.488 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45043.log.gz 2023-10-14T01:30:36.488 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53750.log.gz 2023-10-14T01:30:36.488 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48275.log 2023-10-14T01:30:36.489 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47733.log: /var/log/ceph/ceph-client.admin.45810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47733.log.gz 2023-10-14T01:30:36.489 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45810.log.gz 2023-10-14T01:30:36.489 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48365.log 2023-10-14T01:30:36.489 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40489.log: /var/log/ceph/ceph-client.admin.42375.log: 0.0%/var/log/ceph/ceph.tmp-client.admin.30920.log: -- replaced with /var/log/ceph/ceph-client.admin.52193.log.gz 2023-10-14T01:30:36.490 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40489.log.gz 2023-10-14T01:30:36.490 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42375.log.gz 2023-10-14T01:30:36.490 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52742.log: /var/log/ceph/ceph-client.admin.49155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52742.log.gz 2023-10-14T01:30:36.490 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30920.log.gz 2023-10-14T01:30:36.491 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.48275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47545.log 2023-10-14T01:30:36.491 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49155.log.gz 2023-10-14T01:30:36.491 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48275.log.gz 2023-10-14T01:30:36.491 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53360.log: /var/log/ceph/ceph-client.admin.42123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42123.log.gz 2023-10-14T01:30:36.492 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz 2023-10-14T01:30:36.492 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48365.log.gz 2023-10-14T01:30:36.492 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40017.log 2023-10-14T01:30:36.492 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50372.log: /var/log/ceph/ceph-client.admin.51647.log: /var/log/ceph/ceph-client.admin.44307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54894.log.gz 2023-10-14T01:30:36.493 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50372.log.gz 2023-10-14T01:30:36.493 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51647.log.gz 2023-10-14T01:30:36.493 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44307.log.gz 2023-10-14T01:30:36.493 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52073.log 2023-10-14T01:30:36.494 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34629.log: /var/log/ceph/ceph-client.admin.54656.log: /var/log/ceph/ceph.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2023-10-14T01:30:36.494 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34629.log.gz 2023-10-14T01:30:36.494 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54656.log.gz 2023-10-14T01:30:36.494 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48395.log 2023-10-14T01:30:36.495 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40017.log.gz 2023-10-14T01:30:36.495 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47545.log: /var/log/ceph/ceph-client.admin.52073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47545.log.gz 2023-10-14T01:30:36.495 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52073.log.gz 2023-10-14T01:30:36.495 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40506.log 2023-10-14T01:30:36.496 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51740.log 2023-10-14T01:30:36.496 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48395.log.gz 2023-10-14T01:30:36.496 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40867.log 2023-10-14T01:30:36.497 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40506.log.gz 2023-10-14T01:30:36.497 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40344.log 2023-10-14T01:30:36.497 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51740.log.gz 2023-10-14T01:30:36.497 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52832.log 2023-10-14T01:30:36.498 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40867.log.gz 2023-10-14T01:30:36.498 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50432.log 2023-10-14T01:30:36.498 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40344.log.gz 2023-10-14T01:30:36.498 INFO:teuthology.orchestra.run.smithi105.stderr:gzip/var/log/ceph/ceph-client.admin.52832.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2023-10-14T01:30:36.499 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52832.log.gz 2023-10-14T01:30:36.499 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40000.log 2023-10-14T01:30:36.499 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50432.log.gz 2023-10-14T01:30:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2023-10-14T01:30:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42919.log 2023-10-14T01:30:36.500 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43162.log 2023-10-14T01:30:36.500 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40000.log.gz 2023-10-14T01:30:36.501 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42919.log.gz 2023-10-14T01:30:36.501 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2023-10-14T01:30:36.501 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54551.log 2023-10-14T01:30:36.501 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43162.log.gz 2023-10-14T01:30:36.502 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47793.log 2023-10-14T01:30:36.502 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2023-10-14T01:30:36.502 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54551.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54551.log.gz -5 2023-10-14T01:30:36.503 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43629.log 2023-10-14T01:30:36.503 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47793.log.gz 2023-10-14T01:30:36.503 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2023-10-14T01:30:36.503 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50462.log 2023-10-14T01:30:36.504 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43629.log.gz 2023-10-14T01:30:36.504 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50583.log 2023-10-14T01:30:36.504 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2023-10-14T01:30:36.505 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50462.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.50462.log.gz /var/log/ceph/ceph-client.admin.45936.log 2023-10-14T01:30:36.505 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.505 INFO:teuthology.orchestra.run.smithi105.stderr: 89.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T01:30:36.505 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52862.log 2023-10-14T01:30:36.506 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50583.log.gz 2023-10-14T01:30:36.506 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45936.log.gz 2023-10-14T01:30:36.506 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49733.log 2023-10-14T01:30:36.506 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5/var/log/ceph/ceph-client.admin.52862.log: --verbose -- /var/log/ceph/ceph-client.admin.51436.log 2023-10-14T01:30:36.507 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52862.log.gz 2023-10-14T01:30:36.507 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51132.log 2023-10-14T01:30:36.507 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49733.log.gz 2023-10-14T01:30:36.508 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50096.log 2023-10-14T01:30:36.508 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51436.log.gz 2023-10-14T01:30:36.508 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55160.log 2023-10-14T01:30:36.508 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51132.log.gz 2023-10-14T01:30:36.509 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53857.log 2023-10-14T01:30:36.509 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50096.log.gz 2023-10-14T01:30:36.509 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49794.log 2023-10-14T01:30:36.510 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55160.log.gz 2023-10-14T01:30:36.510 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2023-10-14T01:30:36.510 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53857.log.gz 2023-10-14T01:30:36.510 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49794.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48578.log -- replaced with /var/log/ceph/ceph-client.admin.49794.log.gz 2023-10-14T01:30:36.511 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.511 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51072.log 2023-10-14T01:30:36.511 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2023-10-14T01:30:36.512 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51891.log 2023-10-14T01:30:36.512 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48578.log.gz 2023-10-14T01:30:36.512 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51496.log 2023-10-14T01:30:36.512 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51072.log.gz 2023-10-14T01:30:36.513 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2023-10-14T01:30:36.513 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51891.log.gz 2023-10-14T01:30:36.513 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39143.log 2023-10-14T01:30:36.513 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51496.log.gz 2023-10-14T01:30:36.514 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38888.log 2023-10-14T01:30:36.514 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2023-10-14T01:30:36.514 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54144.log 2023-10-14T01:30:36.514 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39143.log.gz 2023-10-14T01:30:36.515 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38888.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2023-10-14T01:30:36.515 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46059.log 2023-10-14T01:30:36.515 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45228.log 2023-10-14T01:30:36.516 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54144.log.gz 2023-10-14T01:30:36.516 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52315.log 2023-10-14T01:30:36.516 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46059.log.gz 2023-10-14T01:30:36.516 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50613.log 2023-10-14T01:30:36.517 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45228.log.gz 2023-10-14T01:30:36.517 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44899.log 2023-10-14T01:30:36.517 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52315.log.gz 2023-10-14T01:30:36.517 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2023-10-14T01:30:36.518 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50613.log.gz 2023-10-14T01:30:36.518 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44899.log.gz 2023-10-14T01:30:36.518 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54604.log 2023-10-14T01:30:36.519 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54630.log 2023-10-14T01:30:36.519 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2023-10-14T01:30:36.519 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39086.log 2023-10-14T01:30:36.519 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54604.log.gz 2023-10-14T01:30:36.520 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54630.log.gz 2023-10-14T01:30:36.520 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2023-10-14T01:30:36.520 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51679.log 2023-10-14T01:30:36.520 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39086.log.gz 2023-10-14T01:30:36.521 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52557.log 2023-10-14T01:30:36.521 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2023-10-14T01:30:36.521 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51679.log.gz 2023-10-14T01:30:36.522 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34688.log 2023-10-14T01:30:36.522 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55295.log 2023-10-14T01:30:36.522 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52557.log.gz 2023-10-14T01:30:36.522 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44135.log 2023-10-14T01:30:36.523 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34688.log.gz 2023-10-14T01:30:36.523 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53962.log 2023-10-14T01:30:36.523 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55295.log.gz 2023-10-14T01:30:36.523 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50281.log 2023-10-14T01:30:36.524 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44135.log.gz 2023-10-14T01:30:36.524 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53962.log.gz 2023-10-14T01:30:36.524 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2023-10-14T01:30:36.524 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50281.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.50281.log.gz 2023-10-14T01:30:36.525 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54840.log 2023-10-14T01:30:36.525 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51951.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz -5 2023-10-14T01:30:36.525 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43944.log 2023-10-14T01:30:36.526 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43249.log 2023-10-14T01:30:36.526 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54840.log.gz 2023-10-14T01:30:36.526 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47884.log 2023-10-14T01:30:36.526 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43944.log.gz 2023-10-14T01:30:36.527 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54999.log 2023-10-14T01:30:36.527 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43249.log.gz 2023-10-14T01:30:36.527 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47884.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.54338.log -- replaced with /var/log/ceph/ceph-client.admin.47884.log.gz 2023-10-14T01:30:36.527 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.528 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51406.log 2023-10-14T01:30:36.528 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54999.log.gz 2023-10-14T01:30:36.528 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51800.log 2023-10-14T01:30:36.529 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54338.log.gz 2023-10-14T01:30:36.529 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49672.log 2023-10-14T01:30:36.529 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51406.log.gz 2023-10-14T01:30:36.529 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2023-10-14T01:30:36.530 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51800.log.gz 2023-10-14T01:30:36.530 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45691.log 2023-10-14T01:30:36.530 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49672.log.gz 2023-10-14T01:30:36.531 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34756.log 2023-10-14T01:30:36.531 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2023-10-14T01:30:36.531 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45691.log.gz 2023-10-14T01:30:36.531 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38983.log 2023-10-14T01:30:36.532 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39112.log 2023-10-14T01:30:36.532 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34756.log.gz 2023-10-14T01:30:36.532 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.38983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52974.log 2023-10-14T01:30:36.532 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38983.log.gz 2023-10-14T01:30:36.533 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39112.log.gz 2023-10-14T01:30:36.533 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55480.log 2023-10-14T01:30:36.533 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44214.log 2023-10-14T01:30:36.533 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52974.log.gz 2023-10-14T01:30:36.534 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52103.log 2023-10-14T01:30:36.534 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55480.log.gz 2023-10-14T01:30:36.534 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41164.log 2023-10-14T01:30:36.535 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44214.log.gz 2023-10-14T01:30:36.535 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52103.log.gz 2023-10-14T01:30:36.535 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54416.log 2023-10-14T01:30:36.535 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41164.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T01:30:36.536 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41164.log.gz 2023-10-14T01:30:36.536 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53012.log 2023-10-14T01:30:36.536 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54416.log.gz 2023-10-14T01:30:36.536 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48125.log 2023-10-14T01:30:36.537 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40543.log 2023-10-14T01:30:36.537 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53012.log.gz 2023-10-14T01:30:36.537 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53591.log 2023-10-14T01:30:36.538 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48125.log.gz 2023-10-14T01:30:36.538 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41458.log 2023-10-14T01:30:36.538 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40543.log.gz 2023-10-14T01:30:36.538 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2023-10-14T01:30:36.539 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53591.log.gz 2023-10-14T01:30:36.539 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40435.log 2023-10-14T01:30:36.539 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41458.log.gz 2023-10-14T01:30:36.539 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55348.log 2023-10-14T01:30:36.540 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2023-10-14T01:30:36.540 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2023-10-14T01:30:36.540 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40435.log.gz 2023-10-14T01:30:36.540 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53177.log 2023-10-14T01:30:36.541 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55348.log.gz 2023-10-14T01:30:36.541 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52892.log 2023-10-14T01:30:36.541 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz 2023-10-14T01:30:36.542 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54442.log 2023-10-14T01:30:36.542 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53177.log.gz 2023-10-14T01:30:36.542 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45502.log 2023-10-14T01:30:36.542 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52892.log.gz 2023-10-14T01:30:36.543 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53671.log 0.0% 2023-10-14T01:30:36.543 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54442.log.gz 2023-10-14T01:30:36.543 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47854.log 2023-10-14T01:30:36.543 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45502.log.gz 2023-10-14T01:30:36.544 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2023-10-14T01:30:36.544 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53671.log.gz 2023-10-14T01:30:36.544 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43653.log 2023-10-14T01:30:36.544 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47854.log.gz 2023-10-14T01:30:36.545 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2023-10-14T01:30:36.545 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2023-10-14T01:30:36.545 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40688.log 2023-10-14T01:30:36.545 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43653.log.gz 2023-10-14T01:30:36.546 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 2023-10-14T01:30:36.546 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53643.log 2023-10-14T01:30:36.546 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53287.log 2023-10-14T01:30:36.547 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2023-10-14T01:30:36.547 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43309.log 2023-10-14T01:30:36.547 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53643.log.gz 2023-10-14T01:30:36.547 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43680.log 2023-10-14T01:30:36.548 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53287.log.gz 2023-10-14T01:30:36.548 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53121.log 2023-10-14T01:30:36.548 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43309.log.gz 2023-10-14T01:30:36.549 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2023-10-14T01:30:36.549 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43680.log.gz 2023-10-14T01:30:36.549 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46133.log 2023-10-14T01:30:36.549 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53121.log.gz 2023-10-14T01:30:36.550 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54040.log 2023-10-14T01:30:36.550 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2023-10-14T01:30:36.550 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48185.log 2023-10-14T01:30:36.551 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46133.log.gz 2023-10-14T01:30:36.551 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43834.log 2023-10-14T01:30:36.551 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54040.log.gz 2023-10-14T01:30:36.551 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48185.log.gz 2023-10-14T01:30:36.552 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53697.log 2023-10-14T01:30:36.552 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48640.log 2023-10-14T01:30:36.552 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43834.log.gz 2023-10-14T01:30:36.553 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43381.log 2023-10-14T01:30:36.553 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53697.log.gz 2023-10-14T01:30:36.553 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51617.log 2023-10-14T01:30:36.553 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48640.log.gz 2023-10-14T01:30:36.554 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52466.log 2023-10-14T01:30:36.554 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43381.log.gz 2023-10-14T01:30:36.554 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42544.log 2023-10-14T01:30:36.554 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51617.log.gz 2023-10-14T01:30:36.555 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47243.log 2023-10-14T01:30:36.555 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52466.log.gz 2023-10-14T01:30:36.555 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52772.log 2023-10-14T01:30:36.556 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42544.log.gz 2023-10-14T01:30:36.556 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40148.log 2023-10-14T01:30:36.556 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47243.log.gz 2023-10-14T01:30:36.556 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46725.log 2023-10-14T01:30:36.557 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52772.log.gz 2023-10-14T01:30:36.557 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51165.log 2023-10-14T01:30:36.557 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40148.log.gz 2023-10-14T01:30:36.558 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2023-10-14T01:30:36.558 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46725.log.gz 2023-10-14T01:30:36.558 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48065.log 2023-10-14T01:30:36.558 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51165.log.gz 2023-10-14T01:30:36.559 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40614.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.43219.log -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2023-10-14T01:30:36.559 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T01:30:36.559 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51770.log 2023-10-14T01:30:36.560 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48065.log.gz 2023-10-14T01:30:36.560 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2023-10-14T01:30:36.560 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43219.log.gz 2023-10-14T01:30:36.560 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2023-10-14T01:30:36.561 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51770.log.gz 2023-10-14T01:30:36.561 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48884.log 2023-10-14T01:30:36.561 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2023-10-14T01:30:36.562 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34605.log 2023-10-14T01:30:36.562 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2023-10-14T01:30:36.562 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53565.log 2023-10-14T01:30:36.563 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48884.log.gz 2023-10-14T01:30:36.563 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.34605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50674.log 2023-10-14T01:30:36.563 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34605.log.gz 2023-10-14T01:30:36.563 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2023-10-14T01:30:36.564 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53565.log.gz 2023-10-14T01:30:36.564 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47636.log 2023-10-14T01:30:36.564 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50674.log.gz 2023-10-14T01:30:36.564 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2023-10-14T01:30:36.565 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2023-10-14T01:30:36.565 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46910.log 2023-10-14T01:30:36.565 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47636.log.gz 2023-10-14T01:30:36.565 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44381.log 2023-10-14T01:30:36.566 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2023-10-14T01:30:36.566 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44274.log 2023-10-14T01:30:36.566 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46910.log.gz 2023-10-14T01:30:36.567 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53435.log 2023-10-14T01:30:36.567 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44381.log.gz 2023-10-14T01:30:36.567 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47213.log 2023-10-14T01:30:36.567 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44274.log.gz 2023-10-14T01:30:36.568 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55212.log 2023-10-14T01:30:36.568 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53435.log.gz 2023-10-14T01:30:36.568 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46363.log 2023-10-14T01:30:36.568 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47213.log.gz 2023-10-14T01:30:36.569 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54682.log 2023-10-14T01:30:36.569 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55212.log.gz 2023-10-14T01:30:36.569 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54734.log 2023-10-14T01:30:36.569 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46363.log.gz 2023-10-14T01:30:36.570 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54682.log.gz 2023-10-14T01:30:36.570 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48701.log 2023-10-14T01:30:36.570 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42204.log 2023-10-14T01:30:36.570 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54734.log.gz 2023-10-14T01:30:36.571 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40074.log 2023-10-14T01:30:36.571 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48701.log.gz 2023-10-14T01:30:36.571 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42204.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42204.log.gz -5 2023-10-14T01:30:36.572 INFO:teuthology.orchestra.run.smithi105.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49461.log 2023-10-14T01:30:36.572 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43279.log 2023-10-14T01:30:36.572 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40074.log.gz 2023-10-14T01:30:36.572 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54066.log 2023-10-14T01:30:36.573 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49461.log.gz 2023-10-14T01:30:36.573 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46028.log 2023-10-14T01:30:36.573 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43279.log.gz 2023-10-14T01:30:36.574 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42294.log 2023-10-14T01:30:36.574 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54066.log.gz 2023-10-14T01:30:36.574 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45362.log 2023-10-14T01:30:36.574 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46028.log.gz 2023-10-14T01:30:36.575 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52993.log 2023-10-14T01:30:36.575 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42294.log.gz 2023-10-14T01:30:36.575 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46849.log 2023-10-14T01:30:36.575 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45362.log.gz 2023-10-14T01:30:36.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49215.log 2023-10-14T01:30:36.576 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52993.log.gz 2023-10-14T01:30:36.576 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54760.log 0.0% 2023-10-14T01:30:36.577 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46849.log.gz 2023-10-14T01:30:36.577 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48155.log 2023-10-14T01:30:36.577 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49215.log.gz 2023-10-14T01:30:36.577 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55106.log 2023-10-14T01:30:36.578 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54760.log.gz 2023-10-14T01:30:36.578 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48155.log.gz 2023-10-14T01:30:36.578 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40202.log 2023-10-14T01:30:36.578 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50522.log 2023-10-14T01:30:36.579 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55106.log.gz 2023-10-14T01:30:36.579 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46664.log 2023-10-14T01:30:36.579 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40202.log.gz 2023-10-14T01:30:36.580 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50522.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.48005.log 2023-10-14T01:30:36.580 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50522.log.gz 2023-10-14T01:30:36.580 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2023-10-14T01:30:36.580 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46664.log.gz 2023-10-14T01:30:36.581 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2023-10-14T01:30:36.581 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48005.log.gz 2023-10-14T01:30:36.581 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39302.log 2023-10-14T01:30:36.581 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2023-10-14T01:30:36.582 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52436.log 2023-10-14T01:30:36.582 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2023-10-14T01:30:36.582 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.39302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43711.log 2023-10-14T01:30:36.583 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39302.log.gz 2023-10-14T01:30:36.583 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41994.log 2023-10-14T01:30:36.583 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52436.log.gz 2023-10-14T01:30:36.583 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44654.log 2023-10-14T01:30:36.584 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43711.log.gz 2023-10-14T01:30:36.584 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40921.log 2023-10-14T01:30:36.584 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41994.log.gz 2023-10-14T01:30:36.585 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48426.log 2023-10-14T01:30:36.585 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44654.log.gz 2023-10-14T01:30:36.585 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46543.log 2023-10-14T01:30:36.585 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40921.log.gz 2023-10-14T01:30:36.586 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45288.log 2023-10-14T01:30:36.586 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48426.log.gz 2023-10-14T01:30:36.586 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46543.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46543.log.gz 2023-10-14T01:30:36.587 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2023-10-14T01:30:36.587 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42147.log 2023-10-14T01:30:36.587 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45288.log.gz 2023-10-14T01:30:36.587 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47823.log 2023-10-14T01:30:36.588 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2023-10-14T01:30:36.588 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44951.log 2023-10-14T01:30:36.588 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42147.log.gz 2023-10-14T01:30:36.589 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50342.log 2023-10-14T01:30:36.589 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47823.log.gz 2023-10-14T01:30:36.589 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48610.log 2023-10-14T01:30:36.589 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44951.log.gz 2023-10-14T01:30:36.590 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48305.log 2023-10-14T01:30:36.590 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50342.log.gz 2023-10-14T01:30:36.590 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43892.log 2023-10-14T01:30:36.590 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48610.log.gz 2023-10-14T01:30:36.591 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53139.log 2023-10-14T01:30:36.591 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48305.log.gz 2023-10-14T01:30:36.591 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2023-10-14T01:30:36.591 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43892.log.gz 2023-10-14T01:30:36.592 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54578.log 2023-10-14T01:30:36.592 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53139.log.gz 2023-10-14T01:30:36.592 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2023-10-14T01:30:36.592 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2023-10-14T01:30:36.593 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54578.log.gz 2023-10-14T01:30:36.593 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55640.log 2023-10-14T01:30:36.593 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.41290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54364.log 2023-10-14T01:30:36.594 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2023-10-14T01:30:36.594 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50734.log 2023-10-14T01:30:36.594 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55640.log.gz 2023-10-14T01:30:36.594 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2023-10-14T01:30:36.595 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54364.log.gz 2023-10-14T01:30:36.595 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49855.log 2023-10-14T01:30:36.595 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50734.log.gz 2023-10-14T01:30:36.596 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53085.log 2023-10-14T01:30:36.596 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2023-10-14T01:30:36.596 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49581.log 2023-10-14T01:30:36.596 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49855.log.gz 2023-10-14T01:30:36.597 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2023-10-14T01:30:36.597 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53085.log.gz 2023-10-14T01:30:36.597 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52708.log 2023-10-14T01:30:36.597 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49581.log.gz 2023-10-14T01:30:36.598 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42095.log 2023-10-14T01:30:36.598 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.51195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2023-10-14T01:30:36.598 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53513.log 2023-10-14T01:30:36.598 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52708.log.gz 2023-10-14T01:30:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55586.log 2023-10-14T01:30:36.599 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42095.log.gz 2023-10-14T01:30:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2023-10-14T01:30:36.601 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53513.log.gz 2023-10-14T01:30:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.55586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50250.log 2023-10-14T01:30:36.602 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55586.log.gz 2023-10-14T01:30:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46273.log 2023-10-14T01:30:36.602 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2023-10-14T01:30:36.603 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40580.log 2023-10-14T01:30:36.603 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50250.log.gz 2023-10-14T01:30:36.603 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48245.log 2023-10-14T01:30:36.603 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46273.log.gz 2023-10-14T01:30:36.604 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43556.log 2023-10-14T01:30:36.604 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40580.log.gz 2023-10-14T01:30:36.604 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42403.log 2023-10-14T01:30:36.604 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48245.log.gz 2023-10-14T01:30:36.605 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2023-10-14T01:30:36.605 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43556.log.gz 2023-10-14T01:30:36.605 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44796.log 2023-10-14T01:30:36.605 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42403.log.gz 2023-10-14T01:30:36.606 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49430.log 2023-10-14T01:30:36.606 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2023-10-14T01:30:36.606 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43036.log 2023-10-14T01:30:36.607 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44796.log.gz 2023-10-14T01:30:36.607 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50006.log 2023-10-14T01:30:36.607 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49430.log.gz 2023-10-14T01:30:36.607 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48548.log 2023-10-14T01:30:36.608 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43036.log.gz 2023-10-14T01:30:36.608 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2023-10-14T01:30:36.608 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50006.log.gz 2023-10-14T01:30:36.608 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.48548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48548.log.gz 2023-10-14T01:30:36.609 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45196.log 2023-10-14T01:30:36.609 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42670.log 2023-10-14T01:30:36.609 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2023-10-14T01:30:36.609 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40128.log 2023-10-14T01:30:36.610 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45196.log.gz 2023-10-14T01:30:36.610 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.42670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42670.log.gzgzip 2023-10-14T01:30:36.610 INFO:teuthology.orchestra.run.smithi105.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50857.log 2023-10-14T01:30:36.610 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47514.log 2023-10-14T01:30:36.611 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40128.log.gz 2023-10-14T01:30:36.611 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50857.log.gz 2023-10-14T01:30:36.611 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2023-10-14T01:30:36.612 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.47514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46694.log 2023-10-14T01:30:36.612 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47514.log.gz 2023-10-14T01:30:36.612 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.50917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53778.log 2023-10-14T01:30:36.612 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2023-10-14T01:30:36.613 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2023-10-14T01:30:36.613 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46694.log.gz 2023-10-14T01:30:36.613 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45779.log 2023-10-14T01:30:36.614 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53778.log.gz 2023-10-14T01:30:36.614 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53049.log 2023-10-14T01:30:36.614 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gz 2023-10-14T01:30:36.614 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.45779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2023-10-14T01:30:36.615 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45779.log.gz 2023-10-14T01:30:36.615 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.53049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54468.log 2023-10-14T01:30:36.615 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53049.log.gz 2023-10-14T01:30:36.616 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46483.log 2023-10-14T01:30:36.616 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2023-10-14T01:30:36.616 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.54468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44981.log 2023-10-14T01:30:36.617 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54468.log.gz 2023-10-14T01:30:36.617 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.46483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43920.log 2023-10-14T01:30:36.617 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46483.log.gz 2023-10-14T01:30:36.617 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40830.log 2023-10-14T01:30:36.617 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44981.log.gz 2023-10-14T01:30:36.618 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49275.log 2023-10-14T01:30:36.618 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.43920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43920.log.gz 2023-10-14T01:30:36.618 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2023-10-14T01:30:36.619 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40830.log.gz 2023-10-14T01:30:36.619 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.49275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44244.log 2023-10-14T01:30:36.619 INFO:teuthology.orchestra.run.smithi105.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49275.log.gz 2023-10-14T01:30:36.619 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52920.log 2023-10-14T01:30:36.619 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.40418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2023-10-14T01:30:36.620 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.44244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44244.log.gz 2023-10-14T01:30:36.620 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/ceph-client.admin.52920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52920.log.gz 2023-10-14T01:30:36.620 INFO:teuthology.orchestra.run.smithi105.stderr: 90.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T01:30:36.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-10-14T01:30:47.970 INFO:teuthology.orchestra.run.smithi105.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-14T01:30:53.009 INFO:teuthology.orchestra.run.smithi105.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-14T02:03:47.525 INFO:teuthology.orchestra.run.smithi105.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-14T02:15:44.711 INFO:teuthology.orchestra.run.smithi105.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-14T02:15:59.148 INFO:teuthology.orchestra.run.smithi105.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-14T02:15:59.150 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-10-14T02:15:59.150 INFO:teuthology.orchestra.run.smithi105.stderr:real 45m23.059s 2023-10-14T02:15:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:user 42m12.045s 2023-10-14T02:15:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:sys 2m13.064s 2023-10-14T02:15:59.151 INFO:tasks.ceph:Archiving logs... 2023-10-14T02:15:59.153 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274/remote/smithi105/log 2023-10-14T02:15:59.154 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T02:17:56.823 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T02:17:56.840 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-14T02:17:56.840 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T02:17:56.872 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-14T02:17:56.886 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-14T02:17:56.886 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T02:17:57.016 INFO:teuthology.orchestra.run.smithi105.stdout: remote refid st t when poll reach delay offset jitter 2023-10-14T02:17:57.017 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================== 2023-10-14T02:17:57.017 INFO:teuthology.orchestra.run.smithi105.stdout:*hv01.front.sepi 44.4.53.2 2 u 94 256 377 0.189 -4.940 0.570 2023-10-14T02:17:57.018 INFO:teuthology.orchestra.run.smithi105.stdout:+hv02.front.sepi 63.231.80.2 3 u 104 256 377 0.153 -5.628 0.464 2023-10-14T02:17:57.018 INFO:teuthology.orchestra.run.smithi105.stdout:+hv03.front.sepi 74.6.168.72 3 u 126 256 377 0.185 -4.990 0.388 2023-10-14T02:17:57.018 INFO:teuthology.orchestra.run.smithi105.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-14T02:17:57.019 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-14T02:17:57.033 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-14T02:17:57.035 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-14T02:17:57.048 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-14T02:17:57.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-14T02:17:57.085 INFO:teuthology.task.internal:Duration was 8592.250349 seconds 2023-10-14T02:17:57.086 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-14T02:17:57.102 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-14T02:17:57.102 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T02:17:57.472 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-14T02:17:57.473 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2023-10-14T02:17:57.473 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T02:17:57.533 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-14T02:17:57.534 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T02:17:57.638 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-14T02:17:57.639 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T02:17:58.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-14T02:17:58.109 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-14T02:17:58.109 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T02:17:58.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-14T02:17:58.181 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T02:17:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2023-10-14T02:17:58.247 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T02:17:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T02:17:58.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-14T02:17:58.306 INFO:teuthology.task.internal:Transferring archived files... 2023-10-14T02:17:58.308 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274/remote/smithi105 2023-10-14T02:17:58.308 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T02:17:58.360 INFO:teuthology.task.internal:Removing archive directory... 2023-10-14T02:17:58.361 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T02:17:58.408 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-14T02:17:58.426 INFO:teuthology.task.internal:Not uploading archives. 2023-10-14T02:17:58.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-14T02:17:58.438 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-14T02:17:58.439 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T02:17:58.487 INFO:teuthology.orchestra.run.smithi105.stdout: 655472 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 14 02:17 /home/ubuntu/cephtest 2023-10-14T02:17:58.488 INFO:teuthology.orchestra.run.smithi105.stdout: 673495 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 14 00:05 /home/ubuntu/cephtest/mnt.0 2023-10-14T02:17:58.488 INFO:teuthology.orchestra.run.smithi105.stdout: 673920 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 14 00:05 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-14T02:17:58.489 INFO:teuthology.orchestra.run.smithi105.stdout: 687826 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 14 00:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-14T02:17:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T02:17:58.491 INFO:teuthology.orchestra.run.smithi105.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-14T02:17:58.492 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi105 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-14T02:17:58.505 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-14T02:17:58.520 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-14T02:17:58.538 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi105 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-14T02:17:58.538 INFO:teuthology.nuke:Checking targets against current locks 2023-10-14T02:17:58.571 DEBUG:teuthology.nuke:shortname: smithi105 2023-10-14T02:17:58.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T02:17:58.595 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-10-13_23:35:38-rbd-wip-dis-testing-distro-default-smithi/7427274', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-13 23:46:11.581013', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCYvQckp2HFA+3Xt5famVR0+ItJimBVjM4uj2TjVxZTAVlb8a2GoWX8NXDIwcbxdsG5JcnrW4QgTnV7/jDpR5BU='} 2023-10-14T02:17:58.649 INFO:teuthology.orchestra.console.smithi105:Power off 2023-10-14T02:17:58.650 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T02:17:58.677 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2023-10-14T02:17:58.678 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:17:58.703 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2023-10-14T02:18:02.706 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:18:02.816 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2023-10-14T02:18:06.819 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T02:18:06.931 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2023-10-14T02:18:07.032 INFO:teuthology.orchestra.console.smithi105:Power off completed 2023-10-14T02:18:07.138 INFO:teuthology.run:Summary data: description: rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/replicated features/layering msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_support_module_recovery} duration: 8592.250348567963 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi105 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=735f68d7bf91d643813a8420fc2b11441f4bd8c4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_dis@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f97ca0e4b1f64dda8a845dff817b51f0 status: fail success: false 2023-10-14T02:18:07.139 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T02:18:07.227 INFO:teuthology.run:FAIL